Return-Path: X-Original-To: nfsv4@ietfa.amsl.com Delivered-To: nfsv4@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5F3683A0923 for ; Thu, 2 Jul 2020 03:07:00 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N9WIC6Gn6NlR for ; Thu, 2 Jul 2020 03:06:58 -0700 (PDT) Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 97F413A0920 for ; Thu, 2 Jul 2020 03:06:57 -0700 (PDT) Received: by mail-ej1-x633.google.com with SMTP id w6so28626527ejq.6 for ; Thu, 02 Jul 2020 03:06:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=nsLlv/QunOCitlf6rJEIZl6QJggTVoHae3Jlv4pxJu8=; b=JLfGpVM1xeLRUvLb2bl+jGDypmFyW9Z/ww8hcUAGWzKt2rJd31oZIdA8wZcb6b/Y80 B85J3ZSlGXsapUEvu92GjSMnXMu34dMlRRG9Jv/PSKk3n2EFnDjFb2th4T3PdIRIidV2 k91KhytmXm+CyQQZfGmI8pDLWSmZGA0RgXnhqfmhzMUnesoZ7XpR1C6NRYHtDUq4VK36 DOOSo70qh6YfQp6YeHD1x/Z9JK1Npus2sMQXzGcYYxCYm9gemEjzQF83a9aLm8pmgRxS L1AMoAuzJlE2uGS4IxQK25cDBIxiNR+2r8Je+NBYJoPhHLs8dEKKhHdMtbkiJHgPp3dg XFBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=nsLlv/QunOCitlf6rJEIZl6QJggTVoHae3Jlv4pxJu8=; b=WgUsE9yq3p/vqe+JQQLcrTPr4YpVmW6AEUkSLrXu5TDkyjzpqhotXfbAwuj+c4hFRg Aw9uEJ9cnX0nHJtW93fqei/6X1UnOS4Zx6NNZ9Ab2vt1LyBcGzrr4E7BKLby68PUePV6 iUiihI1RJPI9dImvdoRAXcTFlc7bP+j3ImTDTaeSrf1VJCo+PahTFardfCMujCUK+kgO caKqSm4zthoShyty/cJLmCRGQ4PtyS19aP/D6+wA3ZRmA1kRVIVcNaj8iByLxFn6Np62 cQ35e1Y2Ftgpp2Rz5kwsuIv8s0Mz73G5EZeiuvOTu/05Rn0ejePyk49mpZEfIGjCO/KE 6FnA== X-Gm-Message-State: AOAM532tXpegPPPan3YED0kSY+R+cVWYkJhF+f6SnMGQfrRT8kEhnrX3 K27n+z4tQQcbWry3zXgnlRPkdAP/M9N5u9wIZxAt9Q== X-Google-Smtp-Source: ABdhPJzqTZ1nBXHaBBp6xsQ3uhBIgzIrOc6b49LwkGSM6ga42/Sn1Ob3aPkZ3tbHjEWm9VrJOLFUiJlpy9yYZpAffk8= X-Received: by 2002:a17:906:284e:: with SMTP id s14mr25824521ejc.498.1593684416121; Thu, 02 Jul 2020 03:06:56 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a17:906:dbc1:0:0:0:0 with HTTP; Thu, 2 Jul 2020 03:06:55 -0700 (PDT) In-Reply-To: <5d9b4f697ec698a7f07e8168f56826dbb52e234b.camel@hammerspace.com> References: <3fc0af37d7d870eeb6ab854a75d6eeb5aae61a0d.camel@hammerspace.com> <5d9b4f697ec698a7f07e8168f56826dbb52e234b.camel@hammerspace.com> From: David Noveck Date: Thu, 2 Jul 2020 06:06:55 -0400 Message-ID: To: Trond Myklebust Cc: "nfsv4@ietf.org" Content-Type: multipart/alternative; boundary="00000000000081cf8505a972917d" Archived-At: Subject: Re: [nfsv4] Notes regarding discussion of directory scalabiliy issues X-BeenThere: nfsv4@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: NFSv4 Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jul 2020 10:07:00 -0000 --00000000000081cf8505a972917d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tuesday, June 30, 2020, Trond Myklebust wrote: > On Tue, 2020-06-30 at 08:40 -0400, David Noveck wrote: > > Thanks for your helpful comments. > > On Mon, Jun 29, 2020 at 2:43 PM Trond Myklebust > wrote: > > Like it or not, the readdir cookie *is* an attribute of the directory. > > > If the protocol treated them as such, then the attribute notifications > feature could provide updates to the client. Given that it doesn't, we > could add a cookie update feature to directory notification feaure as a > v4.2 extension to the protocol. However, I'm reluctant to start work on > the necessary protocol additions until we are sure they are needed to > provide better directory cacheability. > > Actually, they are attributes of directory *streams. *The difference is > not all that important given that client implementations are unlikely to = be > aware of the specific steam associated with any particular request. > However, there are a few cases in which the difference is important in > determining whether various approaches to client handling of cookies migh= t > or might not work, and will be important in the discussion below: > > - Two requests made on different clients necessarily are made on > distinct streams. > - Two requests made on different instances of the same client (with an > intervening restart/reboot) also have to arise on different streams. > > > If I want to support the POSIX telldir() and seekdir() operations ( > https://pubs.opengroup.org/onlinepubs/9699919799/functions/seekdir.html )= , > then I need to ensure that when the application calls seekdir(), I return > to the exact same cursor location in the stream that I was at when I call= ed > telldir(). > > > Agreed. > > Without a server side cookie on which to anchor my telldir() cookies, > > > Every client has these available but it is not clear to me useful such > anchoring is. I think the flexibility that each client has to > assign cookies to streams it is responsible for is valuable and could be > compromised if anchoring to the server cookies is made the focus of the > implementation. > > then all I have is a list of filenames that can and will change every tim= e > a file is created, deleted or renamed. > > > Clearly it will change. However, the directory notifications feature > makes some assumptions, currently implicit about how the list will change= . > Once these are made explicit, the wg could decide that server/fs pairs > incapable of staying within these reasonable restrictions (if they are, i= n > fact, reasonable), cannot support the directory notifications feature. > > Both the length and ordering of that list may change whenever the > directory is modified, > > > Clearly the length will change, but the reasonable expectation is that > creating a file will increase the length by one and deleting one will > decrease it by one. I don't see the value of supporting directory > notifications on server fs that do something else. > > With regard ro ordering, suppose the spec allows an fs to shuffle the > directory order every time a change is made, but I'm unaware of any actua= l > file systems that do this. Do we need to support directory notification= s > for such fs's? > > > > touch foo; touch bar; ln foo baz; rm foo; mv baz foo > > There... Most filesystems will end up reordering 'foo' and 'bar' in the > directory stream given the above sequence of commands. How does the clien= t > figure out what happened if the above sequence of commands is performed o= n > the server? > Now let's say that is a directory of a million files, and something like > the above is made to happen regularly. How do I maintain a stable list of > synthetic cookies on the client? > > I think you are right about there being cases in which it is impossible, but we either disagree or are simply talking past one another about other cases. If the caching client is making the directory changes, then I agree this cannot be done and you are stuck having to refetch potentially large directories to deal with new READDIR requests=E2=98=B9=EF=B8=8F Where we might disagree is the case in which another client is making the change. In that case directory notifications would allow you to avoid repeated READDIR ops, whether you are providing the user synthetic or server-based cookies. My talk on directory caching will discuss the possibility of v4.2 extensions to address the same-client directory caching issue, as well as possible clarifications regarding directory delegation/notification in v4.1= . > > meaning that a naive implementation > > > OK. I'll plead guilty to one *misdemeanor *count of directory naivety. > > > of synthetic cookies as an offset is not compatible with the > telldir()/seekdir() requirements. > > > It's not clear to me how this incompatibility would manifest itself. I > think I need to understand what would break. > > > To make matters worse, the list size is for all intents and purposes > unbounded, because there is no hard limit on the size of a directory. Tha= t > makes it also impossible to create a cached mapping between a synthetic > cookie and a filename; such a mapping would be unbounded both in size and > in duration (since we don't know a priori how long the application will > keep the directory open, or for that matter, which exact set of cookies i= t > may have cached). > > > Such a mapping would, in essence, be part of the cached directory. So, > if it is too big to keep in client memory,then it is too big to cache and > you might as well decide not to cache it. > > I expect there is an issue that is a worry in the case in which a > reasonably sized directory grows over time to be too big to cache while = an > open directory stream retains some directory cookies which might be > incompatible with the client dropping caching of directories and switchi= ng > to server-based cookies.=F0=9F=98=96 > I feel it is reasonable to treat this situation as one might a > cookie-verifier failure, particularly if this is the only worrisome failu= re > mode. However, this possibility means that I would not ask clients to > implement such local cookies. To enable that, we would have to make > explicit the same sort of reasonableness requirement for cookie changes > that we have already discussed for ordering changes. RFC7530 already > alludes to the need to avoid spurious cookie invalidations although not i= n > as explicit or strict way as we would need to support > directory notifications: > > As there is no way for the client to indicate that a cookie value, > > once received, will not be subsequently used, server implementations > > should avoid schemes that allocate memory corresponding to a returned > > cookie. Such allocation can be avoided if the server bases cookie > > values on a value such as the offset within the directory where the > > scan is to be resumed. > > > Cookies generated by such techniques should be designed to remain > > valid despite modification of the associated directory. If a server > > were to invalidate a cookie because of a directory modification, > > READDIRs of large directories might never finish. > > > > So in order to make this work the client would basically have to create > its own B-tree and persist it in storage somewhere. > > > I don't see the need to make this persistent. If the client restarts, al= l > directory streams have ceased to exist and we know *a posteriori *that > there are no outstanding directory cookies to which the client would have > to respond. > > > > -- > > -- > > Trond Myklebust > Linux NFS client maintainer, Hammerspace > trond.myklebust@hammerspace.com > > > --00000000000081cf8505a972917d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Tuesday, June 30, 2020, Trond Myklebust <trondmy@hammerspace.com> wrote:
On Tue, 2020-06-30 at 08:40 -0400, David Noveck wrote:
Thanks=C2=A0for your helpful comments.

On Mon, Jun 29, 2020 at 2:43 PM Trond= Myklebust <trondmy@hammerspace.com> wrote:
Like it or not, the readdir cookie=C2=A0is=C2=A0an attribute of= the directory.


If the protocol treated=C2=A0them as such, then the attribute notifica= tions feature could provide updates to the client.=C2=A0 =C2=A0Given that i= t doesn't, we could add a cookie update feature to directory notificati= on feaure=C2=A0as a v4.2 extension to the protocol.=C2=A0 However, I'm reluctant to start work on the necessary protocol additions until = we are sure they are=C2=A0needed to provide better=C2=A0directory cacheabil= ity.

Actually, they are attributes of directory streams.=C2=A0 =C2=A0The difference is not all that important=C2=A0given that client implementa= tions are unlikely to be aware=C2=A0of the specific steam associated with a= ny particular request.=C2=A0 However, there are a few cases in=C2=A0which the difference is important=C2=A0in determining whether=C2= =A0various approaches=C2=A0to client handling=C2=A0of cookies might or migh= t not work, and will be important in the discussion=C2=A0below:=C2=A0
  • Two requests made on different clients necessarily are made on distinct= streams.
  • Two requests made on different instances of the same clie= nt (with an intervening restart/reboot) also have to arise on different str= eams.

If I want to support the POSIX telldir() and seekdir() operations (=C2= =A0https://pubs.opengroup.org/onlinepubs/96= 99919799/functions/seekdir.html=C2=A0), then I need to ensure that when the application calls seekdir(), I return to the = exact same cursor location in the stream that I was at when I called telldi= r().


Agreed.=C2=A0

Without a server side cookie on which to anchor my telldir() cookies, =


Every client has these available but it is not clear to me useful such= anchoring is.=C2=A0 =C2=A0I think=C2=A0the flexibility=C2=A0that each clie= nt has to assign=C2=A0cookies to streams it is responsible for is valuable= =C2=A0and could be compromised if anchoring to the server cookies is made the focus of the implementation.

then all I have is a list of filenames that can and will change every = time a file is created, deleted or renamed.


Clearly it will change.=C2=A0 However, the directory notifications fea= ture makes some assumptions, currently implicit about how the list will cha= nge.=C2=A0 =C2=A0Once these are made explicit, the wg could decide that ser= ver/fs pairs incapable of staying=C2=A0within these reasonable restrictions (if they are, in fact, reasonable), cannot support= the directory notifications feature.

Both the length and ordering of that list may change whenever the dire= ctory is modified,


Clearly the length will change, but the reasonable expectation is that= creating a file will increase the length=C2=A0by one and deleting one will= decrease it by one.=C2=A0 =C2=A0I don't see the value of supporting=C2= =A0directory notifications on server fs that do something else.=C2=A0=C2=A0

With regard ro ordering,=C2=A0 suppose the spec allows an fs to shuffl= e the directory order every time a change=C2=A0is made, but I'm unaware= of any actual file systems that do this.=C2=A0 =C2=A0Do we need to support= directory=C2=A0notifications for such fs's?


touch foo; touch bar; ln foo baz; rm foo; mv baz foo

There... Most filesystems will end up reordering 'foo' and = 9;bar' in the directory stream given the above sequence of commands. Ho= w does the client figure out what happened if the above sequence of command= s is performed on the server?
Now let's say that is a directory of a million files, and somethin= g like the above is made to happen regularly. How do I maintain a stable li= st of synthetic cookies on the client?

I think you are right about there being cases= in which it is impossible, but we either disagree or are simply talking pa= st one another about other cases.

If the caching c= lient is making the directory changes, then I agree this cannot be done and= you are stuck having to refetch potentially large directories to deal with= new READDIR requests=E2=98=B9=EF=B8=8F

Where we m= ight disagree is the case in which another client is making the change.=C2= =A0 In that case directory notifications would allow you to avoid repeated = READDIR ops, whether you are providing the user synthetic or server-based c= ookies.

My talk on directory caching will discuss = the possibility of v4.2 extensions to address the same-client directory cac= hing issue, as well as possible clarifications regarding directory delegati= on/notification in v4.1.

=C2=A0
meaning that a naive implementation


OK.=C2=A0 =C2=A0I'll plead guilty to one misdemeanor count = of directory=C2=A0naivety.
=C2=A0
of synthetic cookies as an offset is not compatible with the telldir()= /seekdir() requirements.


It's not clear to me how this incompatibility would manifest itsel= f.=C2=A0 I think I need to understand what would break.
=C2=A0
To make matters worse, the list size is for all intents and purposes u= nbounded, because there is no hard limit on the size of a directory. That m= akes it also impossible to create a cached mapping between a synthetic cook= ie and a filename; such a mapping would be unbounded both in size and in duration (since we don't know a= priori how long the application will keep the directory open, or for that = matter, which exact set of cookies it may have cached).


Such a mapping would, in essence, be part of the=C2=A0cached=C2=A0dire= ctory.=C2=A0 =C2=A0So, if it is=C2=A0too big to keep in client memory,then = it is too big to cache and you might as well decide not to cache=C2=A0it.= =C2=A0=C2=A0

I expect there is an issue that is a worry in the case in which a reas= onably sized=C2=A0directory=C2=A0 grows over time to be too big to cache wh= ile an open directory stream retains some directory cookies which might be = incompatible with the client=C2=A0dropping=C2=A0 caching of directories and switching to server-based cookies.=F0=9F=98=96
I feel it is reasonable to treat this situation as one might a cookie-= verifier failure, particularly if this is the only worrisome=C2=A0failure m= ode.=C2=A0 =C2=A0However, this possibility means that I would not ask clien= ts to implement=C2=A0such local cookies. To enable that, we would have to make explicit the same sort of reasonableness requirement= for cookie changes that we have already=C2=A0discussed for ordering change= s.=C2=A0 RFC7530 already alludes to the need to avoid spurious=C2=A0cookie= =C2=A0invalidations although not in as explicit or strict way as we would need to support directory=C2=A0notifications:
   As there is no way for the client to indicate that a cookie value,<=
/pre>

   once received, will not be subsequently used, server implementation=
s


   should avoid schemes that allocate memory corresponding to a return=
ed


   cookie.  Such allocation can be avoided if the server bases cookie<=
/pre>

   values on a value such as the offset within the directory where the=



   scan is to be resumed.





   Cookies generated by such techniques should be designed to remain

   valid despite modification of the associated directory.  If a serve=
r


   were to invalidate a cookie because of a directory modification,

   READDIRs of large directories might never finish.
=C2=A0
So in order to make this work the client would basically have to creat= e its own B-tree and persist it in storage somewhere.


I don't see the need to make this persistent.=C2=A0 If the client = restarts, all directory streams have ceased to exist and we know=C2=A0 a posteriori=C2=A0 that there are no outstanding=C2=A0directory=C2= =A0cookies=C2=A0to which the client would have to respond.
-- 
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace


--00000000000081cf8505a972917d--