Return-Path: X-Original-To: gen-art@ietfa.amsl.com Delivered-To: gen-art@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 005751A1BB9 for ; Tue, 2 Dec 2014 06:56:45 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.999 X-Spam-Level: X-Spam-Status: No, score=-1.999 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uUNTpsUlbXDL for ; Tue, 2 Dec 2014 06:56:41 -0800 (PST) Received: from mail-ob0-x22a.google.com (mail-ob0-x22a.google.com [IPv6:2607:f8b0:4003:c01::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9B6981A1B8A for ; Tue, 2 Dec 2014 06:56:40 -0800 (PST) Received: by mail-ob0-f170.google.com with SMTP id wp18so9866537obc.29 for ; Tue, 02 Dec 2014 06:56:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=JirRzJuoX45FzPCX/We4i97QJ06ByQbyiyY1uzlJrYY=; b=03tJyrGOnzFb2Lx9WLJqH2KmCeryohbxiE1fjFNkYIbRP5gZ1mymBp3mKXliNv/KBe 7TJjLkQWLXcgdNNREMRB4ck48a9C2ZCWputu0jrFksUsplDQDG4iN+/mz33DrOllHt6m +4Ivf45KvMb7lQnDo0uZ9A/S7NQDAk4Fl8prssf5fSnKPLoZZLokXq0QyqaReOnfg2RU LMXSt/4WyIm7WhmPgsbizmg2ybYnvb3S5OLJbvrZCuBHq8uTySNQoLyb93V1IvLN8b7Z S7E3TUHeB+nCuLmKQmF+qmLqR47c6OO8e2BXO6SAsz7zSPRCSwoyxm3bKuTwJdhVG2Ap 7guA== MIME-Version: 1.0 X-Received: by 10.60.67.165 with SMTP id o5mr40494074oet.24.1417532199865; Tue, 02 Dec 2014 06:56:39 -0800 (PST) Received: by 10.182.5.198 with HTTP; Tue, 2 Dec 2014 06:56:39 -0800 (PST) In-Reply-To: <547DC609.9070104@dial.pipex.com> References: <54627625.9050909@dial.pipex.com> <7E1914F4-413F-4F7E-BE9A-C941FBDD422B@primarydata.com> <5465143D.6020604@dial.pipex.com> <47860B2D-1977-4062-9775-10463F7030B9@primarydata.com> <546B0F4F.3060504@dial.pipex.com> <547C7E2B.10305@dial.pipex.com> <547CC75C.7010909@dial.pipex.com> <1F2B554C-4C7C-4BF0-B078-AA7988F7B0AF@primarydata.com> <547DC609.9070104@dial.pipex.com> Date: Tue, 2 Dec 2014 09:56:39 -0500 Message-ID: From: David Noveck To: Elwyn Davies Content-Type: multipart/alternative; boundary=001a11c31dd83b138b05093cefa2 Archived-At: http://mailarchive.ietf.org/arch/msg/gen-art/HIQGIH_eBvQMXSWZCCXIaYTX5Hs Cc: "gen-art >> General area reviewing team" , Barry Leiba , Tom Haynes Subject: Re: [Gen-art] Gen-art LC review of draft-ietf-nfsv4-rfc3530bis-33 X-BeenThere: gen-art@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "GEN-ART: General Area Review Team" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Dec 2014 14:56:45 -0000 --001a11c31dd83b138b05093cefa2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hence, NFS has no equivalent of the "directory stream" referred to in the Unix manual pages. Furthermore you don't 'open' a directory in NFSv4, you just give it the filehandle. Correct. NFSv4 just took over the NFSv3 approach to reading directories and NFSv3 doesn't support open/close at all. However the lack of a closedir equivalent means that the server has no idea when the client has finished with the READDIR cookie(s). Right. I think it would be helpful to give some advice on what the server should do to avoid a build up of probably useless state. how about: As there is no way for the client to indicate that a cookie value once received, will not be subsequently used, serer implementations should avoid schemes that allocate memory corresponding to a returned cookie. Such allocation can be avoided if the server bases cookie values on a value such as the offset within the directory wher the scan is to be resumed. Given the usual usage of READDIR, I would expect that once a call has returned eof<-TRUE, the client will in almost all cases be done and the state for the (client,directory) pair can be discarded. Except there is no state for client,directory pair. The spec does not mandate that such state be maintained, and i don't know of server that maintain that sort of state. NFSv4. servers follow the NFSv3 practice, where client state* per se* did not exist. However, there might be cases where the call has to be rerun due to comms failure I suppose, so maybe after a number of calls at eof? Presumably there also ought to be some sort of timeout (like if the leases expire??) I have no idea what current implementations do here. None of these situations arise since there is no per--client readdir state. Crafting a couple of sentences to mention these three items (deletion, modification and stale state) would IMO be helpful. It would certainly have helped me!. suggestions: - *deletion* cookies return by READDIR become invalid when the directory is removed but the server is not obliged for check for this situation or check the cookie verifier since the directory handle will become stale. - *modification* Cookies generated by such techniques should reaiin valid despite modification of the associated directory. If a server were to invalidate a cookie because of a directory modification, READDIR's of large directories might never finish. - *stale state* I think it is clear from the text above that there is no state to become stale. Not sure what else you can say. - *directory reorganization* certain re-organization operation on a directory (including directory compaction) may invalidate READDDIR cookies previously given out. when these situation ocur, the server should modify the cookie verifier to accepting cookies which are no longer valid. On Tue, Dec 2, 2014 at 9:00 AM, Elwyn Davies wrote: > Hi. > > Sorry to keep harping on about READDIR, but... > > Tom's comment about opendir/closedir at the end of the mail nbelow > obviously triggered something in my brain overnight, 'cos I woke up > realizing that I wasn't correctly understanding the semantics here. > > Please bear with while I parade my naivete :-( > > Having refreshed my memory on the semantics of reading directories in Uni= x > and checked back on NFSv4, I realized that NFSv4 doesn't have the > equivalents of opendir and closedir. > > Hence, NFS has no equivalent of the "directory stream" referred to in the > Unix manual pages. Furthermore you don't 'open' a directory in NFSv4, yo= u > just give it the filehandle. > > So, the case of the directory getting deleted under your feet between > READDIR calls presumably just results in a NFS4ERR_STALE error on the nex= t > READDIR call and the issue of whether the cookie is valid is moot. It is > then up to the client to map this into local OS behaviour. > > This means that only the directory modification case is relevant to the > cookie validity issue, and I understand the comment that David was making > yesterday more clearly now. > > However the lack of a closedir equivalent means that the server has no > idea when the client has finished with the READDIR cookie(s). I think it > would be helpful to give some advice on what the server should do to avoi= d > a build up of probably useless state. > > Given the usual usage of READDIR, I would expect that once a call has > returned eof<-TRUE, the client will in almost all cases be done and the > state for the (client,directory) pair can be discarded. However, there > might be cases where the call has to be rerun due to comms failure I > suppose, so maybe after a number of calls at eof? Presumably there also > ought to be some sort of timeout (like if the leases expire??) I have no > idea what current implementations do here. > > Crafting a couple of sentences to mention these three items (deletion, > modification and stale state) would IMO be helpful. It would certainly > have helped me! > > Regards, > Elwyn > > On 02/12/14 00:56, Tom Haynes wrote: > >> >> On Dec 1, 2014, at 11:54 AM, Elwyn Davies wrote= : >>> >>> Hi, David. >>> >>> Thanks for the response. >>> >>> A couple of comments below... >>> >>> Regards, >>> Elwyn >>> >>> On 01/12/14 18:50, David Noveck wrote: >>> >>>> My view (remains) that if the cookie is not tied to the change >>>> attribute of the directory, this makes life difficult for both the >>>> server (what does it send if the data has been changed and how doe= s >>>> it manage maintaining checkpoints -- where a readdir finished --, = in >>>> particular if the checkpoint was on an entry that has been deleted= ), >>>> and the client. >>>> >>>> >>>> The cookie is not tied to the change attribute of the directory. >>>> >>>> The purpose of the cookie is ensuring that you do a READDIR that >>>> encompasses multiple requests: >>>> >>>> * entries that exist during the entire period of the multi-requuest >>>> directory read appear exactly once in the result. >>>> * entries that exists during some part of the period of the >>>> multi-request directory read appear at most once in the result. >>>> >>>> >>>> Basically implementations scan through the directory as it exists on >>>> disk and if there is a created entry, the client may or may not see it= , >>>> depending on where the the new entry happens to be put, with respect t= o >>>> the ongoing scan. As far as deleted entries, the implementation only >>>> has to be able to move on to the next entry if it encouneters an entry >>>> marked as deleted. >>>> >>> >>> OK, so the server has to be clever rather than telling the client to >>> restart the readdir -- I'm not sure this is the right trade-off since t= he >>> client has to cope with being told the cached cookie is useless (as per >>> Trond's input below) but I guess implementations work this way already = and >>> it isn't my call. In that case I suggest that the last sentence of >>> s15.26.4, para 9, is modified to tell server implementers a bit more >>> clearly what they have to do. Suggestion: >>> OLD: >>> Ideally, the cookie value should not change if the directory is >>> modified since the client may be caching these values. >>> NEW: >>> The server SHOULD try to accept cookie values issued with READDIR >>> responses even if the directory has been modified between the >>> READDIR calls but MAY return NFS4ERR_NOT_VALID if this is not >>> possible as might be the case if the server has rebooted in the >>> interim. >>> >> >> I=E2=80=99ve taken this change. >> >> >>> One nasty thought occurs to me regarding the idea that a client caches >>> the cookie value(s): Is the server supposed to maintain the whole >>> sequence of cookies for a sequence of READDIRs on a single directory so >>> the client can reissue some of the READDIR calls or is it fair enough >>> for the server to throw them away after the client has used them once? = And >>> how long should the server hang onto cookie state in normal operation? >>> >>> >> No, it does not have to save them. >> >> But the cookies are pretty persistent in the first place. I.e., use some >> property of either the inode or the dirent to generate the cookie. >> >> As such, the cookie state does not need to be held onto. >> >> And for the life, it actually needs to be =E2=80=9Cstable=E2=80=9D betwe= en the opendir() >> and closedir() call on the client. While we are safe on a client reboot,= a >> client could end up holding onto the DIR pointer for a large indetermina= te >> time. >> >> --001a11c31dd83b138b05093cefa2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hen= ce, NFS has no equivalent of the "directory stream" referred to i= n the Unix manual pages.=C2=A0 Furthermore you don't =C2=A0'open' a dir= ectory in NFSv4, you just give it the filehandle.
<= span style=3D"font-family:arial,sans-serif;font-size:13px">
Correct.= =C2=A0 NFSv4 just took over the NFSv3 approach to reading directories and N= FSv3 doesn't support open/close at all. =C2=A0

However the lack of = a closedir equivalent means that the server has no idea when the client has= finished with the READDIR cookie(s).=C2=A0

Right.

=C2=A0I think it would be helpfu= l to give some advice on what the server should do to avoid a build up of p= robably useless state.

how about:

As there is no way for the clien= t to indicate that a cookie value once received, will not be subsequently u= sed, serer implementations should avoid schemes that allocate memory=C2=A0c= orresponding=C2=A0to a returned cookie.=C2=A0 Such allocation can be=C2=A0a= voided if the server bases cookie values on a value such as the offset with= in the directory wher the scan is to be resumed.
<= blockquote style=3D"margin:0px 0px 0px 40px;border:none;padding:0px">
Given the usual usage of READDIR, I = would expect that once a call has returned eof<-TRUE, the client will in= almost all cases be done and the state for the (client,directory) pair can= be discarded. =C2=A0

Except there is = no state for =C2=A0client,directory pair.=C2=A0 The spec does not mandate t= hat such state be maintained, and i don't know of server that maintain = that sort of state.=C2=A0 NFSv4. servers follow the NFSv3 practice, where c= lient state per se did not exist.

However, there might be cases where the c= all has to be rerun due to comms failure I suppose, so maybe after a number= of calls at eof?=C2=A0 Presumably there also ought to be some sort of time= out (like if the leases expire??)=C2=A0 I have no idea what current impleme= ntations do here.

None of these situations arise since there is no per--client readdi= r state.


Crafting a couple of sentences to mention these three item= s (deletion, modification and stale state) would IMO be helpful.=C2=A0 It w= ould certainly have helped me!.
=
suggestions:
  • deletion
    <= /li>
cookies return by READDIR become inval= id when the directory is removed but the server is not obliged for check fo= r this situation or check the cookie verifier since the directory handle wi= ll become stale.
  • modification
Cookies generated by such techniques should reaiin valid despite modi= fication of the associated directory.=C2=A0 If a server were to invalidate = a cookie because of a directory modification, READDIR's of=C2=A0large= =C2=A0directories might never finish.
<= ul>
  • stale state
    • =
    =
    I think it is clear from the text abo= ve that there is no state to become stale.=C2=A0 Not sure what else you can= say.
    • directory reorganization
    certain re-organization operation on a directory (includ= ing directory compaction) may invalidate READDDIR cookies previously=C2=A0g= iven=C2=A0out. =C2=A0when these situation ocur, the server should modify th= e cookie verifier to accepting=C2=A0cookies which are no longer valid.

    On Tue, Dec 2, 2014 at 9:00 AM, Elwyn Davies <elwynd@= dial.pipex.com> wrote:
    Hi.<= br>
    Sorry to keep harping on about READDIR, but...

    Tom's comment about opendir/closedir at the end of the mail nbelow obvi= ously triggered something in my brain overnight, 'cos I woke up realizi= ng that I wasn't correctly understanding the semantics here.

    Please bear with while I parade my naivete :-(

    Having refreshed my memory on the semantics of reading directories in Unix = and checked back on NFSv4, I realized that NFSv4 doesn't have the equiv= alents of opendir and closedir.

    Hence, NFS has no equivalent of the "directory stream" referred t= o in the Unix manual pages.=C2=A0 Furthermore you don't 'open' = a directory in NFSv4, you just give it the filehandle.

    So, the case of the directory getting deleted under your feet between READD= IR calls presumably just results in a NFS4ERR_STALE error on the next READD= IR call and the issue of whether the cookie is valid is moot. It is then up= to the client to map this into local OS behaviour.

    This means that only the directory modification case is relevant to the coo= kie validity issue, and I understand the comment that David was making yest= erday more clearly now.

    However the lack of a closedir equivalent means that the server has no idea= when the client has finished with the READDIR cookie(s).=C2=A0 I think it = would be helpful to give some advice on what the server should do to avoid = a build up of probably useless state.

    Given the usual usage of READDIR, I would expect that once a call has retur= ned eof<-TRUE, the client will in almost all cases be done and the state= for the (client,directory) pair can be discarded.=C2=A0 However, there mig= ht be cases where the call has to be rerun due to comms failure I suppose, = so maybe after a number of calls at eof?=C2=A0 Presumably there also ought = to be some sort of timeout (like if the leases expire??)=C2=A0 I have no id= ea what current implementations do here.

    Crafting a couple of sentences to mention these three items (deletion, modi= fication and stale state) would IMO be helpful.=C2=A0 It would certainly ha= ve helped me!

    Regards,
    Elwyn

    On 02/12/14 00:56, Tom Haynes wrote:

    On Dec 1, 2014, at 11:54 AM, Elwyn Davies <elwynd@dial.pipex.com> wrote:

    Hi, David.

    Thanks for the response.

    A couple of comments below...

    Regards,
    Elwyn

    On 01/12/14 18:50, David Noveck wrote:
    =C2=A0 =C2=A0 My view (remains) that if the cookie is not tied to the chang= e
    =C2=A0 =C2=A0 attribute of the directory, this makes life difficult for bot= h the
    =C2=A0 =C2=A0 server (what does it send if the data has been changed and ho= w does
    =C2=A0 =C2=A0 it manage maintaining checkpoints -- where a readdir finished= --, in
    =C2=A0 =C2=A0 particular if the checkpoint was on an entry that has been de= leted),
    =C2=A0 =C2=A0 and the client.


    The cookie is not tied to the change attribute of the directory.

    The purpose of the cookie is ensuring that you do a READDIR that
    encompasses multiple requests:

    =C2=A0 * entries that exist during the entire period of the multi-requuest<= br> =C2=A0 =C2=A0 directory read appear exactly once in the result.
    =C2=A0 * entries that exists during some part of the period of the
    =C2=A0 =C2=A0 multi-request directory read=C2=A0 appear at most once in the= result.


    Basically implementations scan through the directory as it exists on
    disk and if there is a created entry, the client may or may not see it,
    depending on where the the new entry happens to be put, with respect to
    the ongoing scan.=C2=A0 As far as deleted entries, the implementation only<= br> has to be able to move on to the next entry if it encouneters an entry
    marked as deleted.

    OK, so the server has to be clever rather than telling the client to restar= t the readdir -- I'm not sure this is the right trade-off since the cli= ent has to cope with being told the cached cookie is useless (as per Trond&= #39;s input below) but I guess implementations work this way already and it= isn't my call.=C2=A0 In that case I suggest that the last sentence of = s15.26.4, para 9, is modified to tell server implementers a bit more clearl= y what they have to do.=C2=A0 Suggestion:
    OLD:
    =C2=A0 =C2=A0Ideally, the cookie value should not change if the directory i= s
    =C2=A0 =C2=A0modified since the client may be caching these values.
    NEW:
    =C2=A0 =C2=A0The server SHOULD try to accept cookie values issued with READ= DIR
    =C2=A0 =C2=A0responses even if the directory has been modified between the<= br> =C2=A0 =C2=A0READDIR calls but MAY return NFS4ERR_NOT_VALID if this is not<= br> =C2=A0 =C2=A0possible as might be the case if the server has rebooted in th= e
    =C2=A0 =C2=A0interim.

    I=E2=80=99ve taken this change.


    One nasty thought occurs to me regarding the idea that a client caches
    the cookie value(s):=C2=A0 Is the server supposed to maintain the whole
    sequence of cookies for a sequence of READDIRs on a single directory so
    the client can reissue some of the READDIR calls or is it fair enough for t= he server to throw them away after the client has used them once? And how l= ong should the server hang onto cookie state in normal operation?


    No, it does not have to save them.

    But the cookies are pretty persistent in the first place. I.e., use some pr= operty of either the inode or the dirent to generate the cookie.

    As such, the cookie state does not need to be held onto.

    And for the life, it actually needs to be =E2=80=9Cstable=E2=80=9D between = the opendir() and closedir() call on the client. While we are safe on a cli= ent reboot, a client could end up holding onto the DIR pointer for a large = indeterminate time.


    --001a11c31dd83b138b05093cefa2--