RFC 3530 defines a migration feature which allows the server to direct clients to another server for the purpose of accessing a given file system. While that document explains the feature in terms of a client accessing a given file system and then finding that it has moved, an important limiting case is that in which the clients are redirected as part of their first attempt to access a given file system.
Such redirection is often described as a referral event and implementing such a form of migration has many important consequences, which are not well explained by the presentation within RFC 3530, which often assumes that the client is informed of the migration event after one or more accesses within the file system for which a migration event occurs.
When a client is directed to a new location upon first referencing a file system, the result is best described, from the client's point of view as a referral, rather than a migration event, since the client contains no information derived from the file system before the migration occurred.
Note that the above only refers to a particular client's point of view. A given file system may be accessed by some clients and thus, when a migration occurs, those clients will see an ordinary migration event while other clients see a referral when they first attempt to access the subject filesystem.
In the case in which none of the clients has referenced the subject file system at the time of migration, we have a pure referral situation, in that all that clients will ever see is the referral for an absent file system. Given that clients can use such referrals to find the current location of file systems, servers can usefully provide such referrals when the filesystem in question never actually resided on the server. Such referrals may allow a multi-server namespace to be constructed so that clients may address file systems in terms of the name of the filesystem on a server providing referrals, thereby isolating themselves from server configuration changes which move file systems from server to server.
The details of how referrals proceed are implicit in the specification of migration in RFC 3530. However, because the details of handling of this case are so different from those in the cases discussed therein, examples tailored to the referral situation are needed to clarify matters and allow correct and consistent implementations.
Let us suppose that the following COMPOUND is issued in an environment in which /src/linux/2.7/latest is absent from the target server. This may be for a number of reasons. It may be the case that the file system has moved, or, it may be the case that the target server is functioning mainly or solely to refer clients to the proper server.
Under the given circumstances, the following will be the result.
Current fh is root of pseudo-fs
Current fh is for /src and is within pseudo-fs
Current fh is for /src/linux and is within pseudo-fs
Current fh is for /src/linux/2.7 and is within pseudo-fs
Current fh is for /src/linux/2.7/latest and is within a new, absent fs, but ...
The client will never see the value of that fh
Fails because current fh is in an absent fs at the start of the operation and the spec makes no exception for GETFH.
Not executed because the failure of the GETFH stops processing of the COMPOUND.
Given the failure of the GETFH, the client has the job of determining the root of the absent file system where to find that file system, i.e. the server and path relative to that server's root fh. Note here that while in this example, the client did not obtain filehandles and attribute information (e.g. fsid) for the intermediate directories, he would still not be sure where the absent file system starts. It could be the case, for example, that /src/linux/2.7 is the root of the moved filesystem and that the reason that the lookup of "latest" succeeded is that the filesystem was not absent on that op but was moved between the last LOOKUP and the GETFH (since COMPOUND is not atomic). Even if we had the fsid's for all of the intermediate directories, we could have no way of knowing that /src/linux/2.7/latest was the root of a new fs, since we don't yet have its fsid.
In order to get the necessary information, let us re-issue the chain of lookup's with GETFH's and GETATTR's to at least fsid's and fs-location values.
Current fh is root of pseudo-fs
Just for completeness. Normally, clients will know the fsid of the pseudo-fs as soon as they establish communication with a server.
Get current fsid to see where fs boundaries are. The fsid will be that for the pseudo-fs in this example, so no boundary. Get fs_locations just in case "src" is part of fs that moved.
Current fh is for /src and is within pseudo-fs.
Get current fsid to see where fs boundaries are. The fsid will be that for the pseudo-fs in this example, so no boundary. Get fs_locations just in case "linux" is part of fs that moved.
Current fh is for /src/linux and is within pseudo-fs.
Get current fsid to see where fs boundaries are. The fsid will be that for the pseudo-fs in this example, so no boundary. Get fs_locations just in case "2.7" is part of fs that moved.
Current fh is for /src/linux/2.7 and is within pseudo-fs.
Current fh is for /src/linux/2.7/latest and is within a new, absent fs, but ...
The client will never see the value of that fh
We are getting the fsid to know where the fs boundaries are. The server may oblige us by giving us an fsid value different from that of the pseudo-fs. However, RFC 3530 does not oblige him to give us anything but fs_locations, so this may not be available. Note that if we did have the fsid, it would not necessarily be preserved at the new location. That fsid might be different and in fact the fsid we have for this fs might be the fsid of a different fs on that new server.
In this particular case, we are pretty sure anyway that what has moved is /src/linux/2.7/latest rather than /src/linux/2.7 since we have the fsid of the latter and it is that of the pseudo-fs, which presumably cannot move. However, in other examples, we might not have this kind of information to rely on (e.g. /src/linux/2.7 might be a non-pseudo filesystems separate from /src/linux/2.7/latest), so we need to have another reliable source information on the boundary of the fs which is moved.
The fs_locations attribute indicates the server and server-relative path of the fs's new location but it also gives us the necessary information about the fs boundaries via the fs_root field. In this case the fs_root field /src/lnux/2.7/latest, telling us where the moved fs starts.
Fails because current fh is in an absent fs at the start of the operation and the spec makes no exception for GETFH. Note that this has the happy consequence that we don't have to worry about the volatility or lack thereof of the fh. If the root of the fs on the new location is a persistent fh, then we can assume that this fh, which we never saw is a persistent fh, which, if we could see it, would exactly match the new fh. At least, there is no evidence to disprove that. On the other hand, if we find a volatile root at the new location, then the filehandle which we never saw must have been volatile or at least nobody can prove otherwise.
Given the above, the client knows where the root of the absent file system is, either by noting where the change of fsid occurred, or, if that is not provided, by means of the fs_root field of the fs_locations attribute. The fs_locations attribute also gives the client the actual location of the absent file system, so that the referral can proceed. Generally, the server will give the client the bare minimum of information about the absent file system so that there will be very little scope for problems of conflict between information sent by the referring server and information of the file system's home. No filehandles and very few attributes are present on the referring server and the client can treat those it receives as basically transient information with the function of enabling the referral.
Another context in which a client may encounter referrals is when it does a READDIR on directory in which some of the sub-directories are the roots of absent file systems. In this case, NFS4ERR_MOVED is not an appropriate return because the directory in question is available and only the entries, whose attributes are being interrogated, are for absent file systems.
The appropriate approach for the server in this case is to provide the attributes which it can provide and not provide those that require access to the actual file system, while allowing the client to deduce that an fs boundary is present and that the file system is absent. Fortunately, this can be done fairly easily, as long as the client and server take proper care.
The basic strategy is to return only the attributes that can validly be provided by the referring server. Others are simply not provided. The spec allows this to be done if the client requests the rdattr_error as part of the READDIR, so, if the client does not request this attribute routinely, it must do so when re-issuing a READDIR which gets an NFS4ERR_MOVED error. Without a request for rdattr_error, NFS4ERR_MOVED could mean that the directory being read is within an absent file system or that one or more of the entries in the directory is the root of an absent file system. There is simply no way of determining which unless rdattr_error is requested.
Assuming the rdattr_error is requested, the server should, for the roots of absent file systems, return the attributes listed below when they are requested and no others. Returning other attributes may be possible in particular cases, but generally speaking, they are not necessary for clients to function and since clients will have to be prepared to get the necessary information from the actual root of the fs on the other server, clients are best advised to simply return this small set.
Location of the file system for the client's use when he needs to do the nested mount or to get attribute information about the root of the fs.
Needs to be a value different from the fsid of the containing directory, in order to indicate to the client that a file system boundary is present.
Note that the client should not expect that the actual fs, when located, will have the same fsid. Fsid's are unique only within the set of file systems exported by a single server so it might be possible to maintain the fsid on a different server. In any case, a client will be capable of using file systems on multiple servers and will therefore, if it needs to present unique identifiers for file systems to present to applications, needs to create them and not assume that using fsid's will allow the requisite uniqueness. So a change of mapping from one fsid-server pair to a given id, to another fsid-server pair as part of referral or migration should not pose difficulties.
This is typically needed to return the fileid-related value (i.e. inode number equivalent) for the client system's directory read API. Note that fileid itself will not be available since this is determined on the server where the file system actually is located. If the client needs this value, it must get it from the root of the file system at its new location.
The value should always be NFS4ERR_MOVED for entries that correspond to the root of absent file systems.