Re: [nfsv4] Migration questions

[Chenggong Charles Fan <[e-mail]>]

Hi Dave,

Thanks for the detailed background introduction.  I understand from your 
explanation that using cached file names to do the lookups again after 
migration is problematic when a file name is out of date (e.g. the file 
has been renamed by another client).  It requires addional 
specifications such as the two methods you outlined to make the complete 
case work.

The motivation for standardizing the server-server migration protocol is 
very interesting.  It's a tough problem, especially if the migration is 
to be done transparently to the client, both during the migration and 
after the migration.  Rainfinity has a solution called RainStorage to do 
transparent migration between multi-vendor NFS and CIFS servers.  Let me 
give a quick overview of how it works for NFS.  (By the way, we do have 
a pending parent application for this method of migration).

Let's suppose we'd like to move a directory from filer A to filer B.  
RainStorage is a transparent proxy device that is inserted in front of A 
and B.  Clients continue to mount A or B, without knowledge of the 
existence of the RainStorage box, but RainStorage can see all the NFS 
traffic going to A or B.  RainStorage starts the migration transaction 
by copying files from A to B, and creating an internal fh translation 
table of all the files copied.  For any clients who are accessing any 
files already copied or being copied, the client request is sent to both 
A and B using the fh translation table.  The reply is only sent back to 
the client if received from both A and B.

This way the administrator does not need to shutdown user access at all 
throughout the migration, and at the end, we have a synchronous mirror 
created between A and B for the data that has been migrated, including 
all the user updates that happened during the migration.  This process 
is one way of doing the server-server migration transparently.

Now it comes the question of client cut-over.  Currently for NFS v2 and 
v3, RainStorage cuts over into redirection phase, where for clients who 
are accessing that directory on A, their NFS requests are being 
redirected to B (without their knowledge).  At the same time the global 
namespace map (nowadays, automounter map being the most popular) is 
updated to reflect the new location of the directory (assuming that 
directory is a mount point).  RainStorage can stay in redirection mode 
arbitrarily long, till all clients have auto-unmounted and 
auto-remounted to the destination B.  RainStorage typically stays in 
band for hours.  This process is how we deal with the client-server 
aspect of the migration. 

It would be nice for the client-server part to be built into the v4 
protocol, so that all RainStorage has to do is to update the NFSv4 
namespace server and file server A.  File server A can return 
appropriate error code to the clients (or simply time out), and clients 
can go back to the namespace server to look up the new location, and 
look up their file handles again with the cached file names.

In summary RainStorage is one example how a vendor solution can do the 
server-server part,  i.e. migrate data between mutli-vendor servers, 
within the boundaries of the existing protocol definition.  The 
motivation is to define client-server interactions that make the 
post-migration client cut-over quicker and more transparent.

Thanks,
Charles

Noveck, Dave wrote:

>Let me give you some history on the migration feature in v4.
>
>At the time the migration feature was being thought about, 
>there was active interest in developing a server-to-server
>migration protocol, to allow a smooth multi-vendor transfer
>of the file and directory data, and possibly state and lock
>information, to allow a co-operative transfer of file service
>with minimal impact on the client.
>
>At that time, one school of opinion was that it would be best
>to wait for that effort to complete and then add the migration
>feature to the v4 client-server protocol, in a minor version.
>
>But in thinking about the development and roll-out of such a
>feature, the anticipated timeline looked kind of daunting, to
>those thinking about such an effort as it looked like the
>long pole consisted of development of the server-server protocol,
>followed by development of the extensions to the v4 client-
>server protocol, the implementation of clients capable of 
>dealing with that, and finally the gradual and general deployment
>of clients capable of using the migration feature.  In other words,
>it looked like it would take forever to get something going
>that way.
>
>So the current migration approach was an attempt to get done
>whatever migration support that could be done in clients and 
>servers, without waiting for a server-to-server migration 
>protocol, which was presumed to be on its way.  As things 
>have turned out, that effort did not maintain its momementum
>and ground to a halt.
>
>So, to get back to the client-server world, there were two
>anticipated possibilities.  One was a single-vendor migration
>using persistent filehandles.  A server implementor could
>relatively easily arrange his filehandles so they were unique
>among all machines within the scope of migration.  The filesystem
>could be copied and the old filehandles honored at the new server.
>The idea here was that even though we'd like a muti-vendor 
>solution, enabling the single-vendor solution would get us 
>part of the way there, while work proceeded on the multi-vendor
>approach.
>
>Another approach is use of volatile filehandles but there are
>some issues that restrict use of volatile filehandles for migration.
>It doesn't seem to me that the cost of saving the file name 
>information is one of them.  The first thing to note here is that
>for most file handles you know about, you are caching them and so 
>you do have information that would allow you to have the names for 
>these file handles.  Otherwise, what is the value in caching them?
>However, if you have migration and volatile  file handles, the 
>simplest thing to do with cached file handles is to forget about 
>them.  Caching is an optimization but when you would have to go 
>to the server to re-instantiate, you might as well just wait for 
>a new reference.
>
>There are filehandles for which the client typically has name 
>information and the filehandle cannot be thrown away (e.g. for
>open files), but the troublesome issue is that while it has a name, 
>it might not be correct.   If I open a file, I know a name 
>that the file once had, but with typical Unix semantics I have
>no guarantee that the name has not changed or even that the file
>has any name.  So in that environment, using the name to get a new
>filehandle may get you a different file.  Servers may not allow
>you to change the name of an open file but this is semantics that
>most clients are not expecting and might not like.  My conclusion
>is that volatile file handles for migration, are useful only in
>particular cicumstances.
>
>There are certainly many applications in which they are useful, the
>most common being data which read-only or read mostly.  Dealing
>with archival data, the issue of name change while a file is open may
>not be significant.  In general though, this is an issue, unless people
>are prepared to deal with non-traditional semantics for open.
>
>There has been some discussion of ways to address this issue.  One
>way, in the context of a migration protocol, is for the servers to
>form an fh translation table which the client can interrogate to ask
>the migration destination server what the new file handle is that
>corresponds to a particular old one.
>
>Another way to address this is to allow the client to ask for the 
>current filename of given open file.  Such a call would still be
>valid after the migration event and could be used to get the name
>to be looked up on the new server.  You would need the grace period
>to be applied to rename and delete requests on the new server to
>give the client a guarantee that the new name is still the current
>one.
>
>I think the overall situation is that the current handling of file
>handles in migration is a limited one which is useful in many cases
>but not complete and that some v4 extensions will be required to 
>cleanly address the general case. 
>
>
>
>-----Original Message-----
>From: Chenggong Fan [mailto:[e-mail]]
>Sent: Saturday, October 09, 2004 4:57 PM
>To: [e-mail]
>Subject: [nfsv4] Migration questions
>
>
>Hi,
>
>I have some questions related to the management of file handles after 
>receiving NFS4ERR_MOVED in the migration case (not the pure referral 
>case). 
>
>If a file handle is not a volatile file handle, currently the NFSv4 
>client does not remember the associated pathname with each file handle, 
>correct?  If that's the case, it seems difficult to make the 
>NFS4ERR_MOVED and GETATTR fs_location useful in the migration case.
>
>Let's say the client has been accessing this filesystem for a while, and
>
>suddenly it gets the MOVED error.  Without the associated pathname, how 
>would the client be able to use fs_location information to look up the 
>new file handle in the new file system?   With filehandle being an 
>opaque object, the client shouldn't just replace the fsid field in the 
>file handle.  It seems that for migration to work, all file handles need
>
>to be volatile, (or in other words, client need to keep track of 
>pathnames associated with all the file handles it knows)?   Is it too 
>much a load on the client?
>
>Second, what should the client do to the other file handles that have 
>the same fsid?  Should it not do anything about them?  Or should it 
>expire all of them?  If the first case, the access to each fh will get a
>
>separate MOVED error, and the scenario is the same as the last 
>paragraph.  If this is the case, file-level referral actually might be 
>possible.  If the second case, again the client needed to remember the 
>pathname to look up the new file handles in the new filesystem, or all 
>file handles will be stale.
>
>After all it looks that "pure referral" might be easier to support than 
>migration redirection.  Am I missing anything?
>
>Charles
>
>_______________________________________________
>nfsv4 mailing list
>[e-mail]
>https://www1.ietf.org/mailman/listinfo/nfsv4
>  
>


_______________________________________________
nfsv4 mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/nfsv4