Re: [Gen-art] Gen-art LC review of draft-ietf-nfsv4-rfc3530bis-33

[Thomas Haynes <[e-mail]>]

> On Dec 2, 2014, at 6:00 AM, Elwyn Davies <[e-mail]> wrote:
> 
> Hi.
> 
> Sorry to keep harping on about READDIR, but...
> 
> Tom's comment about opendir/closedir at the end of the mail nbelow obviously triggered something in my brain overnight, 'cos I woke up realizing that I wasn't correctly understanding the semantics here.
> 
> Please bear with while I parade my naivete :-(
> 
> Having refreshed my memory on the semantics of reading directories in Unix and checked back on NFSv4, I realized that NFSv4 doesn't have the equivalents of opendir and closedir.
> 


I was actually talking about on the client, aka opendir(3) and closedir(3) on Linux. So while the protocol may have no notion of these, these do exist and interoperate with the protocol.

In Linux, we have:

const struct file_operations nfs_dir_operations = {
        .llseek         = nfs_llseek_dir,
        .read           = generic_read_dir,
        .iterate        = nfs_readdir,
        .open           = nfs_opendir,
        .release        = nfs_closedir,
        .fsync          = nfs_fsync_dir,
};

I.e., the client is aware of when the closedir(3) occurs.

> Hence, NFS has no equivalent of the "directory stream" referred to in the Unix manual pages.  Furthermore you don't 'open' a directory in NFSv4, you just give it the filehandle.
> 
> So, the case of the directory getting deleted under your feet between READDIR calls presumably just results in a NFS4ERR_STALE error on the next READDIR call and the issue of whether the cookie is valid is moot. It is then up to the client to map this into local OS behaviour.
> 
> This means that only the directory modification case is relevant to the cookie validity issue, and I understand the comment that David was making yesterday more clearly now.
> 
> However the lack of a closedir equivalent means that the server has no idea when the client has finished with the READDIR cookie(s).  I think it would be helpful to give some advice on what the server should do to avoid a build up of probably useless state.
> 
> Given the usual usage of READDIR, I would expect that once a call has returned eof<-TRUE, the client will in almost all cases be done and the state for the (client,directory) pair can be discarded.  However, there might be cases where the call has to be rerun due to comms failure I suppose, so maybe after a number of calls at eof?  Presumably there also ought to be some sort of timeout (like if the leases expire??)  I have no idea what current implementations do here.

This is like any other call to the server which hangs - most likely an EDELAY is passed back up to the caller on the client.


> 
> Crafting a couple of sentences to mention these three items (deletion, modification and stale state) would IMO be helpful.  It would certainly have helped me!
> 
> Regards,
> Elwyn
> 
> On 02/12/14 00:56, Tom Haynes wrote:
>> 
>>> On Dec 1, 2014, at 11:54 AM, Elwyn Davies <[e-mail]> wrote:
>>> 
>>> Hi, David.
>>> 
>>> Thanks for the response.
>>> 
>>> A couple of comments below...
>>> 
>>> Regards,
>>> Elwyn
>>> 
>>> On 01/12/14 18:50, David Noveck wrote:
>>>>    My view (remains) that if the cookie is not tied to the change
>>>>    attribute of the directory, this makes life difficult for both the
>>>>    server (what does it send if the data has been changed and how does
>>>>    it manage maintaining checkpoints -- where a readdir finished --, in
>>>>    particular if the checkpoint was on an entry that has been deleted),
>>>>    and the client.
>>>> 
>>>> 
>>>> The cookie is not tied to the change attribute of the directory.
>>>> 
>>>> The purpose of the cookie is ensuring that you do a READDIR that
>>>> encompasses multiple requests:
>>>> 
>>>>  * entries that exist during the entire period of the multi-requuest
>>>>    directory read appear exactly once in the result.
>>>>  * entries that exists during some part of the period of the
>>>>    multi-request directory read  appear at most once in the result.
>>>> 
>>>> 
>>>> Basically implementations scan through the directory as it exists on
>>>> disk and if there is a created entry, the client may or may not see it,
>>>> depending on where the the new entry happens to be put, with respect to
>>>> the ongoing scan.  As far as deleted entries, the implementation only
>>>> has to be able to move on to the next entry if it encouneters an entry
>>>> marked as deleted.
>>> 
>>> OK, so the server has to be clever rather than telling the client to restart the readdir -- I'm not sure this is the right trade-off since the client has to cope with being told the cached cookie is useless (as per Trond's input below) but I guess implementations work this way already and it isn't my call.  In that case I suggest that the last sentence of s15.26.4, para 9, is modified to tell server implementers a bit more clearly what they have to do.  Suggestion:
>>> OLD:
>>>   Ideally, the cookie value should not change if the directory is
>>>   modified since the client may be caching these values.
>>> NEW:
>>>   The server SHOULD try to accept cookie values issued with READDIR
>>>   responses even if the directory has been modified between the
>>>   READDIR calls but MAY return NFS4ERR_NOT_VALID if this is not
>>>   possible as might be the case if the server has rebooted in the
>>>   interim.
>> 
>> I’ve taken this change.
>> 
>>> 
>>> One nasty thought occurs to me regarding the idea that a client caches
>>> the cookie value(s):  Is the server supposed to maintain the whole
>>> sequence of cookies for a sequence of READDIRs on a single directory so
>>> the client can reissue some of the READDIR calls or is it fair enough for the server to throw them away after the client has used them once? And how long should the server hang onto cookie state in normal operation?
>>> 
>> 
>> No, it does not have to save them.
>> 
>> But the cookies are pretty persistent in the first place. I.e., use some property of either the inode or the dirent to generate the cookie.
>> 
>> As such, the cookie state does not need to be held onto.
>> 
>> And for the life, it actually needs to be “stable” between the opendir() and closedir() call on the client. While we are safe on a client reboot, a client could end up holding onto the DIR pointer for a large indeterminate time.
>>