Return-Path: X-Original-To: gen-art@ietfa.amsl.com Delivered-To: gen-art@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C45F21A1B62 for ; Tue, 2 Dec 2014 10:16:52 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.6 X-Spam-Level: X-Spam-Status: No, score=-2.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Uyak3d6f9mLL for ; Tue, 2 Dec 2014 10:16:44 -0800 (PST) Received: from mail-pd0-f175.google.com (mail-pd0-f175.google.com [209.85.192.175]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5C8881A0392 for ; Tue, 2 Dec 2014 10:15:43 -0800 (PST) Received: by mail-pd0-f175.google.com with SMTP id y10so13603281pdj.6 for ; Tue, 02 Dec 2014 10:15:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=7MkostkFJGJQzZag3XVzJjAPQmyxZ3e5+4nj6o0zb4c=; b=A6HKrQ2JUB51CzEjZrpI2YV7A8QnkB7Ex3NTH4tqASglkCh7bxzYsN9YHGF+uqzUCg UTQwUStPa08KLj82uddhU2FQpp7xZJCga+vlImAviz9Eno22YN4EM07cNBR4to4sxJWm YCVQyjG6dV7ynzfKjvG8cSsKOwvk2QikokXqUByCVK2nNqFX+ebTQT/Q/VDiMfAFNIwA 72ih7W2V07ncSR97XgwM+AytbPQv+sr7+iw/XnxbNSJQyuCvokADWOufHI4SOa9xYHsZ xIPvKE72tVk2nZP9gf9iViYjJQynLZsgYPko1tJZIhpWH25IDcuVSC4yOa2/4j4jG0tK HVVw== X-Gm-Message-State: ALoCoQk1wV7LFcctZ3S5RzLDPBnXtQAs1ooc+auuYYnuu5iiBlnoTNPqW91yLAMGhaLCEIlQmWTH X-Received: by 10.70.130.73 with SMTP id oc9mr1199375pdb.42.1417544142817; Tue, 02 Dec 2014 10:15:42 -0800 (PST) Received: from [10.30.8.37] ([50.242.95.105]) by mx.google.com with ESMTPSA id qd2sm5063219pdb.0.2014.12.02.10.15.41 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 02 Dec 2014 10:15:42 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) From: Tom Haynes In-Reply-To: <547DEBAF.9010907@dial.pipex.com> Date: Tue, 2 Dec 2014 10:15:40 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <0A501549-DC37-43FF-9028-BA0C90514FB9@primarydata.com> References: <54627625.9050909@dial.pipex.com> <7E1914F4-413F-4F7E-BE9A-C941FBDD422B@primarydata.com> <5465143D.6020604@dial.pipex.com> <47860B2D-1977-4062-9775-10463F7030B9@primarydata.com>

<546B0F4F.3060504@dial.pipex.com> <547C7E2B.10305@dial.pipex.com> <547CC75C.7010909@dial.pipex.com> <1F2B554C-4C7C-4BF0-B078-AA7988F7B0AF@primarydata.com> <547DC609.9070104@dial.pipex.com> <547DEBAF.9010907@dial.pipex.com> To: Elwyn Davies X-Mailer: Apple Mail (2.1993) Archived-At: http://mailarchive.ietf.org/arch/msg/gen-art/_5HaKqV54qzLZXpVhDniqL71G6k Cc: "gen-art >> General area reviewing team" , Barry Leiba , David Noveck Subject: Re: [Gen-art] Gen-art LC review of draft-ietf-nfsv4-rfc3530bis-33 X-BeenThere: gen-art@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "GEN-ART: General Area Review Team" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Dec 2014 18:16:55 -0000 Hi Elwyn, Thanks for all of the review and help effort. Besides the inexplicable = typos, you unearthed some issues which would give an experienced reader = pause. Tom > On Dec 2, 2014, at 8:41 AM, Elwyn Davies = wrote: >=20 > Hi, David. >=20 > Excellent. I think some suitable combination of your words below = would fill the bill. >=20 > The point about the very large directories is well taken - I hadn't = grokked that previously. >=20 > I guess the update should go in s15.26.5 after para 1: >=20 > Something like: >=20 > ----------------- >=20 > As there is no way for the client to indicate that a cookie value once = received, will not be subsequently used, server implementations should = avoid schemes that allocate memory corresponding to a returned cookie. = Such allocation can be avoided if the server bases cookie values on a = value such as the offset within the directory where the scan is to be = resumed. >=20 > Cookies generated by such techniques should be designed to remain = valid despite modification of the associated directory. If a server = were to invalidate a cookie because of a directory modification, = READDIR's of large directories might never finish. >=20 > If a directory is deleted after the client has carried out one or more = READDIR operations on the directory, the cookies returned will become = invalid but the server does not need to be concerned as the directory = file handle used previously would have become stale and would be = reported as such on subsequent READDIR operations. The server would not = need to check the cookie verifier in this case. >=20 > However, certain re-organization operations on a directory (including = directory compaction) may invalidate READDDIR cookies previously given = out. When such a situation occurs, the server should modify the cookie = verifier so as to accept cookies which would otherwise no longer be = valid. >=20 Done, with Dave=E2=80=99s suggested tweak. > ------------------- >=20 > Aside: handling reorganization without creating state sounds = challenging but life wouldn't be interesting without challenges! :-) >=20 > A couple of nits that got missed from yesterday: >=20 > s5.4, 2nd bullet labelled '1.': s/common to all object within given = file system./common to all objects within the given file system./ Done >=20 > s9.1.3, last sentence: s/an can be treated as later./and can be = treated as later./ > ^^^^ >=20 Done >=20 > With something like the above, I think we are done. >=20 > Congratulations! I hope this gets to be an RFC RSN. >=20 > Cheers, > Elwyn >=20 >=20 >=20 >=20 >=20 > On 02/12/14 14:56, David Noveck wrote: >> Hence, NFS has no equivalent of the "directory stream" referred to >> in the Unix manual pages. Furthermore you don't 'open' a = directory >> in NFSv4, you just give it the filehandle. >>=20 >>=20 >> Correct. NFSv4 just took over the NFSv3 approach to reading = directories >> and NFSv3 doesn't support open/close at all. >>=20 >> However the lack of a closedir equivalent means that the server = has >> no idea when the client has finished with the READDIR cookie(s). >>=20 >> Right. >>=20 >> I think it would be helpful to give some advice on what the = server >> should do to avoid a build up of probably useless state. >>=20 >> how about: >>=20 >> As there is no way for the client to indicate that a cookie value >> once received, will not be subsequently used, serer = implementations >> should avoid schemes that allocate memory corresponding to a >> returned cookie. Such allocation can be avoided if the server = bases >> cookie values on a value such as the offset within the directory >> wher the scan is to be resumed. >>=20 >>=20 >> Given the usual usage of READDIR, I would expect that once a call >> has returned eof<-TRUE, the client will in almost all cases be = done >> and the state for the (client,directory) pair can be discarded. >>=20 >>=20 >> Except there is no state for client,directory pair. The spec does = not >> mandate that such state be maintained, and i don't know of server = that >> maintain that sort of state. NFSv4. servers follow the NFSv3 = practice, >> where client state/per se/ did not exist. >>=20 >> However, there might be cases where the call has to be rerun due = to >> comms failure I suppose, so maybe after a number of calls at eof? >> Presumably there also ought to be some sort of timeout (like if = the >> leases expire??) I have no idea what current implementations do = here. >>=20 >> None of these situations arise since there is no per--client readdir = state. >>=20 >>=20 >>=20 >> Crafting a couple of sentences to mention these three items = (deletion, >> modification and stale state) would IMO be helpful. It would = certainly >> have helped me!. >>=20 >> suggestions: >>=20 >> * *deletion* >>=20 >> cookies return by READDIR become invalid when the directory is >> removed but the server is not obliged for check for this situation >> or check the cookie verifier since the directory handle will = become >> stale. >>=20 >> * *modification* >>=20 >> Cookies generated by such techniques should reaiin valid despite >> modification of the associated directory. If a server were to >> invalidate a cookie because of a directory modification, READDIR's >> of large directories might never finish. >>=20 >> * *stale state* >>=20 >> I think it is clear from the text above that there is no state to >> become stale. Not sure what else you can say. >>=20 >> * *directory reorganization* >>=20 >> certain re-organization operation on a directory (including >> directory compaction) may invalidate READDDIR cookies >> previously given out. when these situation ocur, the server = should >> modify the cookie verifier to accepting cookies which are no = longer >> valid. >>=20 >>=20 >> On Tue, Dec 2, 2014 at 9:00 AM, Elwyn Davies > > wrote: >>=20 >> Hi. >>=20 >> Sorry to keep harping on about READDIR, but... >>=20 >> Tom's comment about opendir/closedir at the end of the mail nbelow >> obviously triggered something in my brain overnight, 'cos I woke = up >> realizing that I wasn't correctly understanding the semantics = here. >>=20 >> Please bear with while I parade my naivete :-( >>=20 >> Having refreshed my memory on the semantics of reading directories >> in Unix and checked back on NFSv4, I realized that NFSv4 doesn't >> have the equivalents of opendir and closedir. >>=20 >> Hence, NFS has no equivalent of the "directory stream" referred to >> in the Unix manual pages. Furthermore you don't 'open' a = directory >> in NFSv4, you just give it the filehandle. >>=20 >> So, the case of the directory getting deleted under your feet >> between READDIR calls presumably just results in a NFS4ERR_STALE >> error on the next READDIR call and the issue of whether the cookie >> is valid is moot. It is then up to the client to map this into = local >> OS behaviour. >>=20 >> This means that only the directory modification case is relevant = to >> the cookie validity issue, and I understand the comment that David >> was making yesterday more clearly now. >>=20 >> However the lack of a closedir equivalent means that the server = has >> no idea when the client has finished with the READDIR cookie(s). = I >> think it would be helpful to give some advice on what the server >> should do to avoid a build up of probably useless state. >>=20 >> Given the usual usage of READDIR, I would expect that once a call >> has returned eof<-TRUE, the client will in almost all cases be = done >> and the state for the (client,directory) pair can be discarded. >> However, there might be cases where the call has to be rerun due = to >> comms failure I suppose, so maybe after a number of calls at eof? >> Presumably there also ought to be some sort of timeout (like if = the >> leases expire??) I have no idea what current implementations do = here. >>=20 >> Crafting a couple of sentences to mention these three items >> (deletion, modification and stale state) would IMO be helpful. It >> would certainly have helped me! >>=20 >> Regards, >> Elwyn >>=20 >> On 02/12/14 00:56, Tom Haynes wrote: >>=20 >>=20 >> On Dec 1, 2014, at 11:54 AM, Elwyn Davies >> > = wrote: >>=20 >> Hi, David. >>=20 >> Thanks for the response. >>=20 >> A couple of comments below... >>=20 >> Regards, >> Elwyn >>=20 >> On 01/12/14 18:50, David Noveck wrote: >>=20 >> My view (remains) that if the cookie is not tied = to >> the change >> attribute of the directory, this makes life >> difficult for both the >> server (what does it send if the data has been >> changed and how does >> it manage maintaining checkpoints -- where a >> readdir finished --, in >> particular if the checkpoint was on an entry that >> has been deleted), >> and the client. >>=20 >>=20 >> The cookie is not tied to the change attribute of the >> directory. >>=20 >> The purpose of the cookie is ensuring that you do a >> READDIR that >> encompasses multiple requests: >>=20 >> * entries that exist during the entire period of = the >> multi-requuest >> directory read appear exactly once in the result. >> * entries that exists during some part of the = period >> of the >> multi-request directory read appear at most once >> in the result. >>=20 >>=20 >> Basically implementations scan through the directory = as >> it exists on >> disk and if there is a created entry, the client may = or >> may not see it, >> depending on where the the new entry happens to be = put, >> with respect to >> the ongoing scan. As far as deleted entries, the >> implementation only >> has to be able to move on to the next entry if it >> encouneters an entry >> marked as deleted. >>=20 >>=20 >> OK, so the server has to be clever rather than telling the >> client to restart the readdir -- I'm not sure this is the >> right trade-off since the client has to cope with being = told >> the cached cookie is useless (as per Trond's input below) >> but I guess implementations work this way already and it >> isn't my call. In that case I suggest that the last >> sentence of s15.26.4, para 9, is modified to tell server >> implementers a bit more clearly what they have to do. >> Suggestion: >> OLD: >> Ideally, the cookie value should not change if the >> directory is >> modified since the client may be caching these values. >> NEW: >> The server SHOULD try to accept cookie values issued >> with READDIR >> responses even if the directory has been modified >> between the >> READDIR calls but MAY return NFS4ERR_NOT_VALID if this >> is not >> possible as might be the case if the server has = rebooted >> in the >> interim. >>=20 >>=20 >> I=E2=80=99ve taken this change. >>=20 >>=20 >> One nasty thought occurs to me regarding the idea that a >> client caches >> the cookie value(s): Is the server supposed to maintain = the >> whole >> sequence of cookies for a sequence of READDIRs on a single >> directory so >> the client can reissue some of the READDIR calls or is it >> fair enough for the server to throw them away after the >> client has used them once? And how long should the server >> hang onto cookie state in normal operation? >>=20 >>=20 >> No, it does not have to save them. >>=20 >> But the cookies are pretty persistent in the first place. = I.e., >> use some property of either the inode or the dirent to = generate >> the cookie. >>=20 >> As such, the cookie state does not need to be held onto. >>=20 >> And for the life, it actually needs to be =E2=80=9Cstable=E2=80=9D= between the >> opendir() and closedir() call on the client. While we are safe >> on a client reboot, a client could end up holding onto the DIR >> pointer for a large indeterminate time. >>=20 >>=20