Re: [Gen-art] Gen-art LC review of draft-ietf-nfsv4-rfc3530bis-33

[David Noveck <[e-mail]>]

> cookies previously given out.  When such a situation occurs, the server
should modify the cookie verifier so as to
>  accept cookies which would otherwise no longer be vali

''accept cookies'" -->  "disallow use of cookies'

On Tue, Dec 2, 2014 at 11:41 AM, Elwyn Davies <[e-mail]> wrote:

> Hi, David.
>
> Excellent.  I think some suitable combination of your words below would
> fill the bill.
>
> The point about the very large directories is well taken - I hadn't
> grokked that previously.
>
> I guess the update should go in s15.26.5 after para 1:
>
> Something like:
>
> -----------------
>
> As there is no way for the client to indicate that a cookie value once
> received, will not be subsequently used, server implementations should
> avoid schemes that allocate memory corresponding to a returned cookie. Such
> allocation can be avoided if the server bases cookie values on a value such
> as the offset within the directory where the scan is to be resumed.
>
> Cookies generated by such techniques should be designed to remain valid
> despite modification of the associated directory.  If a server were to
> invalidate a cookie because of a directory modification, READDIR's of large
> directories might never finish.
>
> If a directory is deleted after the client has carried out one or more
> READDIR operations on the directory, the cookies returned will become
> invalid but the server does not need to be concerned as the directory file
> handle used previously would have become stale and would be reported as
> such on subsequent READDIR operations.  The server would not need to check
> the cookie verifier in this case.
>
> However, certain re-organization operations on a directory (including
> directory compaction) may invalidate READDDIR cookies previously given
> out.  When such a situation occurs, the server should modify the cookie
> verifier so as to accept cookies which would otherwise no longer be valid.
>
> -------------------
>
> Aside: handling reorganization without creating state sounds challenging
> but life wouldn't be interesting without challenges! :-)
>
> A couple of nits that got missed from yesterday:
>
> s5.4, 2nd bullet labelled '1.': s/common to all object within given file
> system./common to all objects within the given file system./
>
> s9.1.3, last sentence: s/an can be treated as later./and can be treated as
> later./
>                          ^^^^
>
>
> With something like the above, I think we are done.
>
> Congratulations!  I hope this gets to be an RFC RSN.
>
> Cheers,
> Elwyn
>
>
>
>
>
> On 02/12/14 14:56, David Noveck wrote:
>
>>     Hence, NFS has no equivalent of the "directory stream" referred to
>>     in the Unix manual pages.  Furthermore you don't 'open' a directory
>>     in NFSv4, you just give it the filehandle.
>>
>>
>> Correct.  NFSv4 just took over the NFSv3 approach to reading directories
>> and NFSv3 doesn't support open/close at all.
>>
>>     However the lack of a closedir equivalent means that the server has
>>     no idea when the client has finished with the READDIR cookie(s).
>>
>> Right.
>>
>>       I think it would be helpful to give some advice on what the server
>>     should do to avoid a build up of probably useless state.
>>
>> how about:
>>
>>     As there is no way for the client to indicate that a cookie value
>>     once received, will not be subsequently used, serer implementations
>>     should avoid schemes that allocate memory corresponding to a
>>     returned cookie.  Such allocation can be avoided if the server bases
>>     cookie values on a value such as the offset within the directory
>>     wher the scan is to be resumed.
>>
>>
>>     Given the usual usage of READDIR, I would expect that once a call
>>     has returned eof<-TRUE, the client will in almost all cases be done
>>     and the state for the (client,directory) pair can be discarded.
>>
>>
>> Except there is no state for  client,directory pair.  The spec does not
>> mandate that such state be maintained, and i don't know of server that
>> maintain that sort of state.  NFSv4. servers follow the NFSv3 practice,
>> where client state/per se/ did not exist.
>>
>>     However, there might be cases where the call has to be rerun due to
>>     comms failure I suppose, so maybe after a number of calls at eof?
>>     Presumably there also ought to be some sort of timeout (like if the
>>     leases expire??)  I have no idea what current implementations do here.
>>
>> None of these situations arise since there is no per--client readdir
>> state.
>>
>>
>>
>> Crafting a couple of sentences to mention these three items (deletion,
>> modification and stale state) would IMO be helpful.  It would certainly
>> have helped me!.
>>
>> suggestions:
>>
>>   * *deletion*
>>
>>     cookies return by READDIR become invalid when the directory is
>>     removed but the server is not obliged for check for this situation
>>     or check the cookie verifier since the directory handle will become
>>     stale.
>>
>>   * *modification*
>>
>>     Cookies generated by such techniques should reaiin valid despite
>>     modification of the associated directory.  If a server were to
>>     invalidate a cookie because of a directory modification, READDIR's
>>     of large directories might never finish.
>>
>>   * *stale state*
>>
>>     I think it is clear from the text above that there is no state to
>>     become stale.  Not sure what else you can say.
>>
>>   * *directory reorganization*
>>
>>     certain re-organization operation on a directory (including
>>     directory compaction) may invalidate READDDIR cookies
>>     previously given out.  when these situation ocur, the server should
>>     modify the cookie verifier to accepting cookies which are no longer
>>     valid.
>>
>>
>> On Tue, Dec 2, 2014 at 9:00 AM, Elwyn Davies <[e-mail]
>> <mailto:[e-mail]>> wrote:
>>
>>     Hi.
>>
>>     Sorry to keep harping on about READDIR, but...
>>
>>     Tom's comment about opendir/closedir at the end of the mail nbelow
>>     obviously triggered something in my brain overnight, 'cos I woke up
>>     realizing that I wasn't correctly understanding the semantics here.
>>
>>     Please bear with while I parade my naivete :-(
>>
>>     Having refreshed my memory on the semantics of reading directories
>>     in Unix and checked back on NFSv4, I realized that NFSv4 doesn't
>>     have the equivalents of opendir and closedir.
>>
>>     Hence, NFS has no equivalent of the "directory stream" referred to
>>     in the Unix manual pages.  Furthermore you don't 'open' a directory
>>     in NFSv4, you just give it the filehandle.
>>
>>     So, the case of the directory getting deleted under your feet
>>     between READDIR calls presumably just results in a NFS4ERR_STALE
>>     error on the next READDIR call and the issue of whether the cookie
>>     is valid is moot. It is then up to the client to map this into local
>>     OS behaviour.
>>
>>     This means that only the directory modification case is relevant to
>>     the cookie validity issue, and I understand the comment that David
>>     was making yesterday more clearly now.
>>
>>     However the lack of a closedir equivalent means that the server has
>>     no idea when the client has finished with the READDIR cookie(s).  I
>>     think it would be helpful to give some advice on what the server
>>     should do to avoid a build up of probably useless state.
>>
>>     Given the usual usage of READDIR, I would expect that once a call
>>     has returned eof<-TRUE, the client will in almost all cases be done
>>     and the state for the (client,directory) pair can be discarded.
>>     However, there might be cases where the call has to be rerun due to
>>     comms failure I suppose, so maybe after a number of calls at eof?
>>     Presumably there also ought to be some sort of timeout (like if the
>>     leases expire??)  I have no idea what current implementations do here.
>>
>>     Crafting a couple of sentences to mention these three items
>>     (deletion, modification and stale state) would IMO be helpful.  It
>>     would certainly have helped me!
>>
>>     Regards,
>>     Elwyn
>>
>>     On 02/12/14 00:56, Tom Haynes wrote:
>>
>>
>>             On Dec 1, 2014, at 11:54 AM, Elwyn Davies
>>             <[e-mail] <mailto:[e-mail]>> wrote:
>>
>>             Hi, David.
>>
>>             Thanks for the response.
>>
>>             A couple of comments below...
>>
>>             Regards,
>>             Elwyn
>>
>>             On 01/12/14 18:50, David Noveck wrote:
>>
>>                      My view (remains) that if the cookie is not tied to
>>                 the change
>>                      attribute of the directory, this makes life
>>                 difficult for both the
>>                      server (what does it send if the data has been
>>                 changed and how does
>>                      it manage maintaining checkpoints -- where a
>>                 readdir finished --, in
>>                      particular if the checkpoint was on an entry that
>>                 has been deleted),
>>                      and the client.
>>
>>
>>                 The cookie is not tied to the change attribute of the
>>                 directory.
>>
>>                 The purpose of the cookie is ensuring that you do a
>>                 READDIR that
>>                 encompasses multiple requests:
>>
>>                    * entries that exist during the entire period of the
>>                 multi-requuest
>>                      directory read appear exactly once in the result.
>>                    * entries that exists during some part of the period
>>                 of the
>>                      multi-request directory read  appear at most once
>>                 in the result.
>>
>>
>>                 Basically implementations scan through the directory as
>>                 it exists on
>>                 disk and if there is a created entry, the client may or
>>                 may not see it,
>>                 depending on where the the new entry happens to be put,
>>                 with respect to
>>                 the ongoing scan.  As far as deleted entries, the
>>                 implementation only
>>                 has to be able to move on to the next entry if it
>>                 encouneters an entry
>>                 marked as deleted.
>>
>>
>>             OK, so the server has to be clever rather than telling the
>>             client to restart the readdir -- I'm not sure this is the
>>             right trade-off since the client has to cope with being told
>>             the cached cookie is useless (as per Trond's input below)
>>             but I guess implementations work this way already and it
>>             isn't my call.  In that case I suggest that the last
>>             sentence of s15.26.4, para 9, is modified to tell server
>>             implementers a bit more clearly what they have to do.
>>             Suggestion:
>>             OLD:
>>                 Ideally, the cookie value should not change if the
>>             directory is
>>                 modified since the client may be caching these values.
>>             NEW:
>>                 The server SHOULD try to accept cookie values issued
>>             with READDIR
>>                 responses even if the directory has been modified
>>             between the
>>                 READDIR calls but MAY return NFS4ERR_NOT_VALID if this
>>             is not
>>                 possible as might be the case if the server has rebooted
>>             in the
>>                 interim.
>>
>>
>>         I’ve taken this change.
>>
>>
>>             One nasty thought occurs to me regarding the idea that a
>>             client caches
>>             the cookie value(s):  Is the server supposed to maintain the
>>             whole
>>             sequence of cookies for a sequence of READDIRs on a single
>>             directory so
>>             the client can reissue some of the READDIR calls or is it
>>             fair enough for the server to throw them away after the
>>             client has used them once? And how long should the server
>>             hang onto cookie state in normal operation?
>>
>>
>>         No, it does not have to save them.
>>
>>         But the cookies are pretty persistent in the first place. I.e.,
>>         use some property of either the inode or the dirent to generate
>>         the cookie.
>>
>>         As such, the cookie state does not need to be held onto.
>>
>>         And for the life, it actually needs to be “stable” between the
>>         opendir() and closedir() call on the client. While we are safe
>>         on a client reboot, a client could end up holding onto the DIR
>>         pointer for a large indeterminate time.
>>
>>
>>