Re: [Gen-art] Gen-art LC review of draft-ietf-nfsv4-rfc3530bis-33

[Tom Haynes <[e-mail]>]

Hi Elwyn,

Thanks for all of the review and help effort. Besides the inexplicable typos, you unearthed some issues which would give an experienced reader pause.

Tom


> On Dec 2, 2014, at 8:41 AM, Elwyn Davies <[e-mail]> wrote:
> 
> Hi, David.
> 
> Excellent.  I think some suitable combination of your words below would fill the bill.
> 
> The point about the very large directories is well taken - I hadn't grokked that previously.
> 
> I guess the update should go in s15.26.5 after para 1:
> 
> Something like:
> 
> -----------------
> 
> As there is no way for the client to indicate that a cookie value once received, will not be subsequently used, server implementations should avoid schemes that allocate memory corresponding to a returned cookie. Such allocation can be avoided if the server bases cookie values on a value such as the offset within the directory where the scan is to be resumed.
> 
> Cookies generated by such techniques should be designed to remain valid despite modification of the associated directory.  If a server were to invalidate a cookie because of a directory modification, READDIR's of large directories might never finish.
> 
> If a directory is deleted after the client has carried out one or more READDIR operations on the directory, the cookies returned will become invalid but the server does not need to be concerned as the directory file handle used previously would have become stale and would be reported as such on subsequent READDIR operations.  The server would not need to check the cookie verifier in this case.
> 
> However, certain re-organization operations on a directory (including directory compaction) may invalidate READDDIR cookies previously given out.  When such a situation occurs, the server should modify the cookie verifier so as to accept cookies which would otherwise no longer be valid.
> 

Done, with Dave’s suggested tweak.


> -------------------
> 
> Aside: handling reorganization without creating state sounds challenging but life wouldn't be interesting without challenges! :-)
> 
> A couple of nits that got missed from yesterday:
> 
> s5.4, 2nd bullet labelled '1.': s/common to all object within given file system./common to all objects within the given file system./

Done


> 
> s9.1.3, last sentence: s/an can be treated as later./and can be treated as later./
>                         ^^^^
> 

Done

> 
> With something like the above, I think we are done.
> 
> Congratulations!  I hope this gets to be an RFC RSN.
> 
> Cheers,
> Elwyn
> 
> 
> 
> 
> 
> On 02/12/14 14:56, David Noveck wrote:
>>    Hence, NFS has no equivalent of the "directory stream" referred to
>>    in the Unix manual pages.  Furthermore you don't 'open' a directory
>>    in NFSv4, you just give it the filehandle.
>> 
>> 
>> Correct.  NFSv4 just took over the NFSv3 approach to reading directories
>> and NFSv3 doesn't support open/close at all.
>> 
>>    However the lack of a closedir equivalent means that the server has
>>    no idea when the client has finished with the READDIR cookie(s).
>> 
>> Right.
>> 
>>      I think it would be helpful to give some advice on what the server
>>    should do to avoid a build up of probably useless state.
>> 
>> how about:
>> 
>>    As there is no way for the client to indicate that a cookie value
>>    once received, will not be subsequently used, serer implementations
>>    should avoid schemes that allocate memory corresponding to a
>>    returned cookie.  Such allocation can be avoided if the server bases
>>    cookie values on a value such as the offset within the directory
>>    wher the scan is to be resumed.
>> 
>> 
>>    Given the usual usage of READDIR, I would expect that once a call
>>    has returned eof<-TRUE, the client will in almost all cases be done
>>    and the state for the (client,directory) pair can be discarded.
>> 
>> 
>> Except there is no state for  client,directory pair.  The spec does not
>> mandate that such state be maintained, and i don't know of server that
>> maintain that sort of state.  NFSv4. servers follow the NFSv3 practice,
>> where client state/per se/ did not exist.
>> 
>>    However, there might be cases where the call has to be rerun due to
>>    comms failure I suppose, so maybe after a number of calls at eof?
>>    Presumably there also ought to be some sort of timeout (like if the
>>    leases expire??)  I have no idea what current implementations do here.
>> 
>> None of these situations arise since there is no per--client readdir state.
>> 
>> 
>> 
>> Crafting a couple of sentences to mention these three items (deletion,
>> modification and stale state) would IMO be helpful.  It would certainly
>> have helped me!.
>> 
>> suggestions:
>> 
>>  * *deletion*
>> 
>>    cookies return by READDIR become invalid when the directory is
>>    removed but the server is not obliged for check for this situation
>>    or check the cookie verifier since the directory handle will become
>>    stale.
>> 
>>  * *modification*
>> 
>>    Cookies generated by such techniques should reaiin valid despite
>>    modification of the associated directory.  If a server were to
>>    invalidate a cookie because of a directory modification, READDIR's
>>    of large directories might never finish.
>> 
>>  * *stale state*
>> 
>>    I think it is clear from the text above that there is no state to
>>    become stale.  Not sure what else you can say.
>> 
>>  * *directory reorganization*
>> 
>>    certain re-organization operation on a directory (including
>>    directory compaction) may invalidate READDDIR cookies
>>    previously given out.  when these situation ocur, the server should
>>    modify the cookie verifier to accepting cookies which are no longer
>>    valid.
>> 
>> 
>> On Tue, Dec 2, 2014 at 9:00 AM, Elwyn Davies <[e-mail]
>> <mailto:[e-mail]>> wrote:
>> 
>>    Hi.
>> 
>>    Sorry to keep harping on about READDIR, but...
>> 
>>    Tom's comment about opendir/closedir at the end of the mail nbelow
>>    obviously triggered something in my brain overnight, 'cos I woke up
>>    realizing that I wasn't correctly understanding the semantics here.
>> 
>>    Please bear with while I parade my naivete :-(
>> 
>>    Having refreshed my memory on the semantics of reading directories
>>    in Unix and checked back on NFSv4, I realized that NFSv4 doesn't
>>    have the equivalents of opendir and closedir.
>> 
>>    Hence, NFS has no equivalent of the "directory stream" referred to
>>    in the Unix manual pages.  Furthermore you don't 'open' a directory
>>    in NFSv4, you just give it the filehandle.
>> 
>>    So, the case of the directory getting deleted under your feet
>>    between READDIR calls presumably just results in a NFS4ERR_STALE
>>    error on the next READDIR call and the issue of whether the cookie
>>    is valid is moot. It is then up to the client to map this into local
>>    OS behaviour.
>> 
>>    This means that only the directory modification case is relevant to
>>    the cookie validity issue, and I understand the comment that David
>>    was making yesterday more clearly now.
>> 
>>    However the lack of a closedir equivalent means that the server has
>>    no idea when the client has finished with the READDIR cookie(s).  I
>>    think it would be helpful to give some advice on what the server
>>    should do to avoid a build up of probably useless state.
>> 
>>    Given the usual usage of READDIR, I would expect that once a call
>>    has returned eof<-TRUE, the client will in almost all cases be done
>>    and the state for the (client,directory) pair can be discarded.
>>    However, there might be cases where the call has to be rerun due to
>>    comms failure I suppose, so maybe after a number of calls at eof?
>>    Presumably there also ought to be some sort of timeout (like if the
>>    leases expire??)  I have no idea what current implementations do here.
>> 
>>    Crafting a couple of sentences to mention these three items
>>    (deletion, modification and stale state) would IMO be helpful.  It
>>    would certainly have helped me!
>> 
>>    Regards,
>>    Elwyn
>> 
>>    On 02/12/14 00:56, Tom Haynes wrote:
>> 
>> 
>>            On Dec 1, 2014, at 11:54 AM, Elwyn Davies
>>            <[e-mail] <mailto:[e-mail]>> wrote:
>> 
>>            Hi, David.
>> 
>>            Thanks for the response.
>> 
>>            A couple of comments below...
>> 
>>            Regards,
>>            Elwyn
>> 
>>            On 01/12/14 18:50, David Noveck wrote:
>> 
>>                     My view (remains) that if the cookie is not tied to
>>                the change
>>                     attribute of the directory, this makes life
>>                difficult for both the
>>                     server (what does it send if the data has been
>>                changed and how does
>>                     it manage maintaining checkpoints -- where a
>>                readdir finished --, in
>>                     particular if the checkpoint was on an entry that
>>                has been deleted),
>>                     and the client.
>> 
>> 
>>                The cookie is not tied to the change attribute of the
>>                directory.
>> 
>>                The purpose of the cookie is ensuring that you do a
>>                READDIR that
>>                encompasses multiple requests:
>> 
>>                   * entries that exist during the entire period of the
>>                multi-requuest
>>                     directory read appear exactly once in the result.
>>                   * entries that exists during some part of the period
>>                of the
>>                     multi-request directory read  appear at most once
>>                in the result.
>> 
>> 
>>                Basically implementations scan through the directory as
>>                it exists on
>>                disk and if there is a created entry, the client may or
>>                may not see it,
>>                depending on where the the new entry happens to be put,
>>                with respect to
>>                the ongoing scan.  As far as deleted entries, the
>>                implementation only
>>                has to be able to move on to the next entry if it
>>                encouneters an entry
>>                marked as deleted.
>> 
>> 
>>            OK, so the server has to be clever rather than telling the
>>            client to restart the readdir -- I'm not sure this is the
>>            right trade-off since the client has to cope with being told
>>            the cached cookie is useless (as per Trond's input below)
>>            but I guess implementations work this way already and it
>>            isn't my call.  In that case I suggest that the last
>>            sentence of s15.26.4, para 9, is modified to tell server
>>            implementers a bit more clearly what they have to do.
>>            Suggestion:
>>            OLD:
>>                Ideally, the cookie value should not change if the
>>            directory is
>>                modified since the client may be caching these values.
>>            NEW:
>>                The server SHOULD try to accept cookie values issued
>>            with READDIR
>>                responses even if the directory has been modified
>>            between the
>>                READDIR calls but MAY return NFS4ERR_NOT_VALID if this
>>            is not
>>                possible as might be the case if the server has rebooted
>>            in the
>>                interim.
>> 
>> 
>>        I’ve taken this change.
>> 
>> 
>>            One nasty thought occurs to me regarding the idea that a
>>            client caches
>>            the cookie value(s):  Is the server supposed to maintain the
>>            whole
>>            sequence of cookies for a sequence of READDIRs on a single
>>            directory so
>>            the client can reissue some of the READDIR calls or is it
>>            fair enough for the server to throw them away after the
>>            client has used them once? And how long should the server
>>            hang onto cookie state in normal operation?
>> 
>> 
>>        No, it does not have to save them.
>> 
>>        But the cookies are pretty persistent in the first place. I.e.,
>>        use some property of either the inode or the dirent to generate
>>        the cookie.
>> 
>>        As such, the cookie state does not need to be held onto.
>> 
>>        And for the life, it actually needs to be “stable” between the
>>        opendir() and closedir() call on the client. While we are safe
>>        on a client reboot, a client could end up holding onto the DIR
>>        pointer for a large indeterminate time.
>> 
>>