Return-Path: X-Original-To: nfsv4@ietfa.amsl.com Delivered-To: nfsv4@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EDAAB3A0C27 for ; Fri, 3 Jul 2020 03:43:58 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3mK1sHkFdsDw for ; Fri, 3 Jul 2020 03:43:56 -0700 (PDT) Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 824AE3A0C25 for ; Fri, 3 Jul 2020 03:43:55 -0700 (PDT) Received: by mail-ed1-x533.google.com with SMTP id dm19so21016303edb.13 for ; Fri, 03 Jul 2020 03:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=2I5Gjwnoj92wLA6TMo9sFBwcInUYq8l+dHUg1BKqOKo=; b=qeRFBfHS5anaJSj02Z6EXpHK4zzb/nVcD8OYbtfLJ0+uIyVDMQbM6Skjgxm2pfZn/8 gUR1buGabA6a9/hCb1FbfvVflOsaP3KejYh4EmFd4zWlMVozPTeYfW0jCMUnRWaMUyOz M++0VIToo526+fdadW2ipb2qNDANM15wlxxA5bSOaFwb85xKZFTvk+CNdkAfMYnCZVF9 73kzVvUwweOm7QFk/toRU87S3S1oLYJpZj5pCfBvtd0klpWZrkJAvE9MkqsLi+H/+b6f ybWRb+dyE9PHAL5bZyTquTiFfnEvjjTEIk0Qc+MTVy3rF9DQQP4umYBSdXhiP1FeuQ1u OtmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=2I5Gjwnoj92wLA6TMo9sFBwcInUYq8l+dHUg1BKqOKo=; b=djrOPvEtr/irdzK9vEWwocs67vGuL9aJgK5IGaGiesqmOd64irlUYweTcHRZEMK2wk +HBts2ZxL2XgM6G7GVkFLt5UahDE811pM6PjvF+R8N8pHKMBbHqTTny/lXtm02Hx6vAv KNadLAOn4G/NJAcfqvfeiY3RIqRjfRviIAWmQDnE7uI7lPAqUvySE1hlpu4FNHnMT4eO R2Es3bBaf2CfP3ozPCF3+lvbnXLO9hCX18vWNWBNIj0jmFXat1qrG2q4Jaiyhf8j9AuR 6as5d4qz4xAHgDYroApQp/NU0mKiWIxGX4/u75IiVIun6tyKvT5YV328wAa64VMBJxQd IzAw== X-Gm-Message-State: AOAM533GjTstcojgyERDSVZL2bD+EfK2MDgAXAyjcClZzsxu7VnMzy2Z SWpWNLGUdNq2EetnfTIOb72phDIFg53IYdgGk2p9dQ== X-Google-Smtp-Source: ABdhPJxsBs5MqHQZxBYcvCcAcWvCWxbDC0dAeAbVGYeLwJL62i3xa+YEoMVMMNptZq/I1X3b5vwvNy8FYB0Sjvki8VE= X-Received: by 2002:a50:f702:: with SMTP id g2mr40202926edn.348.1593773033891; Fri, 03 Jul 2020 03:43:53 -0700 (PDT) MIME-Version: 1.0 References: <3fc0af37d7d870eeb6ab854a75d6eeb5aae61a0d.camel@hammerspace.com> <5d9b4f697ec698a7f07e8168f56826dbb52e234b.camel@hammerspace.com> In-Reply-To: From: David Noveck Date: Fri, 3 Jul 2020 06:43:42 -0400 Message-ID: To: Rick Macklem Cc: Trond Myklebust , NFSv4 Content-Type: multipart/alternative; boundary="00000000000089a94a05a9873313" Archived-At: Subject: Re: [nfsv4] Notes regarding discussion of directory scalabiliy issues X-BeenThere: nfsv4@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: NFSv4 Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jul 2020 10:43:59 -0000 --00000000000089a94a05a9873313 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Jul 2, 2020, 9:03 PM Rick Macklem wrote: > Well, I'll throw out a simple suggestion w.r.t. directory cookies. > I'm also working on a suggestion about helping clients maintain directory cookies. Slides will be presented 7/9. > What if the cookies generated by the server were required to have the > following properties: > I don't think we can add new requirements for directory cookies. Requirements for the granting or retention of directory delegations are possible but I'd like to derive them from inherent aspects of the existing notification scheme. Adding wholly new requirements on an existing feature is not possible. - Monotonically increasing numeric values. > Not possible to add this requirment. Also don't see the need/value of this. In my view, the existing directory notification scheme provides adequate information for subscribed clients to maintain the structure of cached directories being modified, including cookie values. - Sparse enough that new entries can be created with numeric values > between them. > Again, a new requirement, with no clear justification. - Guaranteed to not change and remain valid (and referring to the same > entry) > despite additions/deletions. (Until a deletion callback is received by > the client.) > Should also be allowed if the directory delegation is recalled. I'm unclear about the callbacks you are proposing. Normally, notification callbacks are not sent to the client making the change. It might be possible to change that in a v4.2 extension but it adds to operation latency and creates sequencing difficulties =E2=98=B9=EF=B8=8F I considered returning the necessary info in a bunch of new ops but the whole thing got waaay too complicated. I think the best choice would be a GET_NOTIFICATION op to be issued after the CREATE, OPEN, LINK, REMOVE, RENAME. > For example, for a simple case of a UFS file system on a server, the UFS > directory consists of blocks of directory entries. > - When an entry is added, it goes in a gap within the directory or grows = a > new block at the end (if I vaguely recall this correctly;-). > - When an entry is removed, the entry is erased without moving other > entries. > For this example, the directory offset cookie can simply be the byte offs= et > of the entry. > (I haven't looked at other file systems, but hopefully offset cookies wit= h > the > above properties can be created for most of them?) > Possibly, but given the need for this, I'm not inclined to rely on hope. > > When an entry is added/deleted, the server issues a callback to the clien= t > with the new directory cookie offset (and the directory entry for the add > case). > I'd prefer that the same information that is returned as in the existing directory notification scheme, which allows you to maintain directory order and cookies, without tieing one to the other. > The client can maintain this structure in any number of ways (and it coul= d > be > fun figuring out what works well), but a trivial version could be: > - The reply to a readdir is kept as a list head (with the directory offse= t > of the > first entry) and a linked list of the entries in order, with their > cookie. > (Remember that the cookies are in the same ordering as the entries > Not in my scheme. - The next readdir reply creates the next head/list. > > readdir() just works through each list, following each head in order. > telldir() returns the cookie for the entry. > seekdir() just finds the correct list and then searches down that list fo= r > a match. > > The remove/add entry callbacks just insert/delete entrie(s) in the > appropriate > list. > (I'd probably keep these lists in the kernel client under the VFS for > FreeBSD, > as malloc'd data structures, but that is simply an implementation choice= .) > > You would require the monotonically increasing property for directory > delegations to be issued. (Without that you don't know where to insert > additions.) The insert notification gives you enough information to do this without the monotonically increasing property. You would also require that extant directory entry cookies > remain valid and unchanged when additions/deletions occur. > (Note that removing an entry and then adding an entry at the same offset > is allowed under POSIX telldir()/seekdir() as I understand it.) > I think the requirement goes away when the delegation does. > This avoids any need for the client to synthesize cookies and just use th= e > ones > returned by the server, I think? > Yes > Just a simple idea that may be worth considering? > (If this has already been discussed, Don't think it has. I apologize for not seeing it.) > Even if it had, no apology would be necessary > rick > > ________________________________________ > From: nfsv4 on behalf of David Noveck < > davenoveck@gmail.com> > Sent: Thursday, July 2, 2020 6:06 AM > To: Trond Myklebust > Cc: nfsv4@ietf.org > Subject: Re: [nfsv4] Notes regarding discussion of directory scalabiliy > issues > > CAUTION: This email originated from outside of the University of Guelph. > Do not click links or open attachments unless you recognize the sender an= d > know the content is safe. If in doubt, forward suspicious emails to > IThelp@uoguelph.ca > > > > On Tuesday, June 30, 2020, Trond Myklebust > wrote: > On Tue, 2020-06-30 at 08:40 -0400, David Noveck wrote: > Thanks for your helpful comments. > > On Mon, Jun 29, 2020 at 2:43 PM Trond Myklebust > wrote: > Like it or not, the readdir cookie is an attribute of the directory. > > > If the protocol treated them as such, then the attribute notifications > feature could provide updates to the client. Given that it doesn't, we > could add a cookie update feature to directory notification feaure as a > v4.2 extension to the protocol. However, I'm reluctant to start work on > the necessary protocol additions until we are sure they are needed to > provide better directory cacheability. > > Actually, they are attributes of directory streams. The difference is > not all that important given that client implementations are unlikely to = be > aware of the specific steam associated with any particular request. > However, there are a few cases in which the difference is important in > determining whether various approaches to client handling of cookies migh= t > or might not work, and will be important in the discussion below: > > * Two requests made on different clients necessarily are made on > distinct streams. > * Two requests made on different instances of the same client (with a= n > intervening restart/reboot) also have to arise on different streams. > > If I want to support the POSIX telldir() and seekdir() operations ( > https://pubs.opengroup.org/onlinepubs/9699919799/functions/seekdir.html > ), then I need to ensure that when the application calls seekdir(), I > return to the exact same cursor location in the stream that I was at when= I > called telldir(). > > > Agreed. > > Without a server side cookie on which to anchor my telldir() cookies, > > > Every client has these available but it is not clear to me useful such > anchoring is. I think the flexibility that each client has to assign > cookies to streams it is responsible for is valuable and could be > compromised if anchoring to the server cookies is made the focus of the > implementation. > > then all I have is a list of filenames that can and will change every tim= e > a file is created, deleted or renamed. > > > Clearly it will change. However, the directory notifications feature > makes some assumptions, currently implicit about how the list will change= . > Once these are made explicit, the wg could decide that server/fs pairs > incapable of staying within these reasonable restrictions (if they are, i= n > fact, reasonable), cannot support the directory notifications feature. > > Both the length and ordering of that list may change whenever the > directory is modified, > > > Clearly the length will change, but the reasonable expectation is that > creating a file will increase the length by one and deleting one will > decrease it by one. I don't see the value of supporting directory > notifications on server fs that do something else. > > With regard ro ordering, suppose the spec allows an fs to shuffle the > directory order every time a change is made, but I'm unaware of any actua= l > file systems that do this. Do we need to support directory notification= s > for such fs's? > > > touch foo; touch bar; ln foo baz; rm foo; mv baz foo > > There... Most filesystems will end up reordering 'foo' and 'bar' in the > directory stream given the above sequence of commands. How does the clien= t > figure out what happened if the above sequence of commands is performed o= n > the server? > Now let's say that is a directory of a million files, and something like > the above is made to happen regularly. How do I maintain a stable list of > synthetic cookies on the client? > > I think you are right about there being cases in which it is impossible, > but we either disagree or are simply talking past one another about other > cases. > > If the caching client is making the directory changes, then I agree this > cannot be done and you are stuck having to refetch potentially large > directories to deal with new READDIR requests=E2=98=B9=EF=B8=8F > > Where we might disagree is the case in which another client is making the > change. In that case directory notifications would allow you to avoid > repeated READDIR ops, whether you are providing the user synthetic or > server-based cookies. > > My talk on directory caching will discuss the possibility of v4.2 > extensions to address the same-client directory caching issue, as well as > possible clarifications regarding directory delegation/notification in v4= .1. > > > meaning that a naive implementation > > > OK. I'll plead guilty to one misdemeanor count of directory naivety. > > of synthetic cookies as an offset is not compatible with the > telldir()/seekdir() requirements. > > > It's not clear to me how this incompatibility would manifest itself. I > think I need to understand what would break. > > To make matters worse, the list size is for all intents and purposes > unbounded, because there is no hard limit on the size of a directory. Tha= t > makes it also impossible to create a cached mapping between a synthetic > cookie and a filename; such a mapping would be unbounded both in size and > in duration (since we don't know a priori how long the application will > keep the directory open, or for that matter, which exact set of cookies i= t > may have cached). > > > Such a mapping would, in essence, be part of the cached directory. So, > if it is too big to keep in client memory,then it is too big to cache and > you might as well decide not to cache it. > > I expect there is an issue that is a worry in the case in which a > reasonably sized directory grows over time to be too big to cache while = an > open directory stream retains some directory cookies which might be > incompatible with the client dropping caching of directories and switchi= ng > to server-based cookies.=F0=9F=98=96 > I feel it is reasonable to treat this situation as one might a > cookie-verifier failure, particularly if this is the only worrisome failu= re > mode. However, this possibility means that I would not ask clients to > implement such local cookies. To enable that, we would have to make > explicit the same sort of reasonableness requirement for cookie changes > that we have already discussed for ordering changes. RFC7530 already > alludes to the need to avoid spurious cookie invalidations although not i= n > as explicit or strict way as we would need to support directory > notifications: > > As there is no way for the client to indicate that a cookie value, > > once received, will not be subsequently used, server implementations > > should avoid schemes that allocate memory corresponding to a returned > > cookie. Such allocation can be avoided if the server bases cookie > > values on a value such as the offset within the directory where the > > scan is to be resumed. > > > Cookies generated by such techniques should be designed to remain > > valid despite modification of the associated directory. If a server > > were to invalidate a cookie because of a directory modification, > > READDIRs of large directories might never finish. > > > So in order to make this work the client would basically have to create > its own B-tree and persist it in storage somewhere. > > > I don't see the need to make this persistent. If the client restarts, al= l > directory streams have ceased to exist and we know a posteriori that > there are no outstanding directory cookies to which the client would have > to respond. > > > -- > > -- > > Trond Myklebust > Linux NFS client maintainer, Hammerspace > trond.myklebust@hammerspace.com > > > > --00000000000089a94a05a9873313 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Thu, Jul 2, 2020, 9:03 PM Rick Macklem <rmacklem@uoguelph.ca&= gt; wrote:
Well, I'll throw out= a simple suggestion w.r.t. directory cookies.
=

I'm also working on a sug= gestion about helping clients maintain directory cookies. Slides will be pr= esented 7/9.


What if the cookies generated by the server were required to have the
following properties:

I don't think we can add new requirements for dire= ctory cookies.=C2=A0 Requirements for the granting or retention of director= y delegations are possible but I'd like=C2=A0 to derive them from inher= ent aspects of the existing notification scheme. Adding wholly new requirem= ents on an existing feature is not possible.

- Monotonically increasing numeric values.

Not possible to add this requirme= nt. Also don't see the need/value of this.=C2=A0 In my view, the existi= ng directory notification scheme provides adequate information for subscrib= ed clients to maintain the structure of cached directories being modified, = including cookie values.

- Sparse enough that new entries can be created with numeric values
=C2=A0 between them.

Again, a new requirement, with no clear justification.<= /div>

- Guaranteed to not change and remain valid (and referring to the same entr= y)
=C2=A0 despite additions/deletions. (Until a deletion callback is received = by the client.)

Should also be allowed if the directory delegation is recall= ed.

I'm unclear abou= t the callbacks you are proposing. Normally, notification callbacks are not= sent to the client making the change. It might be possible to change that = in a v4.2 extension but it adds to operation latency and creates sequencing= difficulties =E2=98=B9=EF=B8=8F

I considered returning the necessary info in a bunch of new ops b= ut the whole thing got waaay too complicated.=C2=A0 I think the best choice= would be a GET_NOTIFICATION op to be issued after the CREATE, OPEN, LINK, = REMOVE, RENAME.


For example, for a simple case of a UFS file system on a server, the UFS directory consists of blocks of directory entries.
- When an entry is added, it goes in a gap within the directory or grows a<= br> =C2=A0 =C2=A0new block at the end (if I vaguely recall this correctly;-). - When an entry is removed, the entry is erased without moving other
=C2=A0 =C2=A0entries.
For this example, the directory offset cookie can simply be the byte offset=
of the entry.
(I haven't looked at other file systems, but hopefully offset cookies w= ith the
=C2=A0above properties can be created for most of them?)

Possibly, but given= the need for this, I'm not inclined to rely on hope.

When an entry is added/deleted, the server issues a callback to the client<= br> with the new directory cookie offset (and the directory entry for the add c= ase).

I'd prefer that the same information that is returned as in the ex= isting directory notification scheme, which allows you to maintain director= y order and cookies, without tieing one to the other.


The client can maintain this structure in any number of ways (and it could = be
fun figuring out what works well), but a trivial version could be:
- The reply to a readdir is kept as a list head (with the directory offset = of the
=C2=A0 =C2=A0first entry) and a linked list of the entries in order, with t= heir cookie.
=C2=A0 (Remember that the cookies are in the same ordering as the entries

No= t in my scheme.

- The next readdir reply creates the next head/list.

readdir() just works through each list, following each head in order.
telldir() returns the cookie for the entry.
seekdir() just finds the correct list and then searches down that list for = a match.

The remove/add entry callbacks just insert/delete entrie(s) in the appropri= ate
list.
(I'd probably keep these lists in the kernel client under the VFS for F= reeBSD,
=C2=A0as malloc'd data structures, but that is simply an implementation= choice.)

You would require the monotonically increasing property for directory
delegations to be issued. (Without that you don't know where to insert<= br> additions.)

The insert notification gives you enough information to do this w= ithout the monotonically increasing property.

You would also require that extant directory entry cookies
remain valid and unchanged when additions/deletions occur.
(Note that removing an entry and then adding an entry at the same offset =C2=A0is allowed under POSIX telldir()/seekdir() as I understand it.)

I thin= k the requirement goes away when the delegation does.


This avoids any need for the client to synthesize cookies and just use the = ones
returned by the server, I think?

Yes


Just a simple idea that may be worth considering?
(If this has already been discussed,

Don't think it has.

I apologize for not seeing it.)

Even if it had, no apology w= ould be necessary


rick

________________________________________
From: nfsv4 <nfsv4-bounces@ietf.org> on behalf of David Noveck <davenovec= k@gmail.com>
Sent: Thursday, July 2, 2020 6:06 AM
To: Trond Myklebust
Cc: nfsv4@iet= f.org
Subject: Re: [nfsv4] Notes regarding discussion of directory scalabiliy iss= ues

CAUTION: This email originated from outside of the University of Guelph. Do= not click links or open attachments unless you recognize the sender and kn= ow the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.= ca



On Tuesday, June 30, 2020, Trond Myklebust <trondmy@hammerspace.com<m= ailto:trondmy@hammerspace.com>> wrote:
On Tue, 2020-06-30 at 08:40 -0400, David Noveck wrote:
Thanks for your helpful comments.

On Mon, Jun 29, 2020 at 2:43 PM Trond Myklebust <trondmy@hammerspace.com= <mailto:trondmy@hammerspace.com>> wrote:
Like it or not, the readdir cookie is an attribute of the directory.


If the protocol treated them as such, then the attribute notifications feat= ure could provide updates to the client.=C2=A0 =C2=A0Given that it doesn= 9;t, we could add a cookie update feature to directory notification feaure = as a v4.2 extension to the protocol.=C2=A0 However, I'm reluctant to st= art work on the necessary protocol additions until we are sure they are nee= ded to provide better directory cacheability.

Actually, they are attributes of directory streams.=C2=A0 =C2=A0The differe= nce is not all that important given that client implementations are unlikel= y to be aware of the specific steam associated with any particular request.= =C2=A0 However, there are a few cases in which the difference is important = in determining whether various approaches to client handling of cookies mig= ht or might not work, and will be important in the discussion below:

=C2=A0 *=C2=A0 =C2=A0Two requests made on different clients necessarily are= made on distinct streams.
=C2=A0 *=C2=A0 =C2=A0Two requests made on different instances of the same c= lient (with an intervening restart/reboot) also have to arise on different = streams.

If I want to support the POSIX telldir() and seekdir() operations ( https://pubs.opengroup.org/onlinep= ubs/9699919799/functions/seekdir.html ), then I need to ensure that whe= n the application calls seekdir(), I return to the exact same cursor locati= on in the stream that I was at when I called telldir().


Agreed.

Without a server side cookie on which to anchor my telldir() cookies,


Every client has these available but it is not clear to me useful such anch= oring is.=C2=A0 =C2=A0I think the flexibility that each client has to assig= n cookies to streams it is responsible for is valuable and could be comprom= ised if anchoring to the server cookies is made the focus of the implementa= tion.

then all I have is a list of filenames that can and will change every time = a file is created, deleted or renamed.


Clearly it will change.=C2=A0 However, the directory notifications feature = makes some assumptions, currently implicit about how the list will change.= =C2=A0 =C2=A0Once these are made explicit, the wg could decide that server/= fs pairs incapable of staying within these reasonable restrictions (if they= are, in fact, reasonable), cannot support the directory notifications feat= ure.

Both the length and ordering of that list may change whenever the directory= is modified,


Clearly the length will change, but the reasonable expectation is that crea= ting a file will increase the length by one and deleting one will decrease = it by one.=C2=A0 =C2=A0I don't see the value of supporting directory no= tifications on server fs that do something else.

With regard ro ordering,=C2=A0 suppose the spec allows an fs to shuffle the= directory order every time a change is made, but I'm unaware of any ac= tual file systems that do this.=C2=A0 =C2=A0Do we need to support directory= notifications for such fs's?


touch foo; touch bar; ln foo baz; rm foo; mv baz foo

There... Most filesystems will end up reordering 'foo' and 'bar= ' in the directory stream given the above sequence of commands. How doe= s the client figure out what happened if the above sequence of commands is = performed on the server?
Now let's say that is a directory of a million files, and something lik= e the above is made to happen regularly. How do I maintain a stable list of= synthetic cookies on the client?

I think you are right about there being cases in which it is impossible, bu= t we either disagree or are simply talking past one another about other cas= es.

If the caching client is making the directory changes, then I agree this ca= nnot be done and you are stuck having to refetch potentially large director= ies to deal with new READDIR requests=E2=98=B9=EF=B8=8F

Where we might disagree is the case in which another client is making the c= hange.=C2=A0 In that case directory notifications would allow you to avoid = repeated READDIR ops, whether you are providing the user synthetic or serve= r-based cookies.

My talk on directory caching will discuss the possibility of v4.2 extension= s to address the same-client directory caching issue, as well as possible c= larifications regarding directory delegation/notification in v4.1.


meaning that a naive implementation


OK.=C2=A0 =C2=A0I'll plead guilty to one misdemeanor count of directory= naivety.

of synthetic cookies as an offset is not compatible with the telldir()/seek= dir() requirements.


It's not clear to me how this incompatibility would manifest itself.=C2= =A0 I think I need to understand what would break.

To make matters worse, the list size is for all intents and purposes unboun= ded, because there is no hard limit on the size of a directory. That makes = it also impossible to create a cached mapping between a synthetic cookie an= d a filename; such a mapping would be unbounded both in size and in duratio= n (since we don't know a priori how long the application will keep the = directory open, or for that matter, which exact set of cookies it may have = cached).


Such a mapping would, in essence, be part of the cached directory.=C2=A0 = =C2=A0So, if it is too big to keep in client memory,then it is too big to c= ache and you might as well decide not to cache it.

I expect there is an issue that is a worry in the case in which a reasonabl= y sized directory=C2=A0 grows over time to be too big to cache while an ope= n directory stream retains some directory cookies which might be incompatib= le with the client dropping=C2=A0 caching of directories and switching to s= erver-based cookies.=F0=9F=98=96
I feel it is reasonable to treat this situation as one might a cookie-verif= ier failure, particularly if this is the only worrisome failure mode.=C2=A0= =C2=A0However, this possibility means that I would not ask clients to impl= ement such local cookies. To enable that, we would have to make explicit th= e same sort of reasonableness requirement for cookie changes that we have a= lready discussed for ordering changes.=C2=A0 RFC7530 already alludes to the= need to avoid spurious cookie invalidations although not in as explicit or= strict way as we would need to support directory notifications:

=C2=A0 =C2=A0As there is no way for the client to indicate that a cookie va= lue,

=C2=A0 =C2=A0once received, will not be subsequently used, server implement= ations

=C2=A0 =C2=A0should avoid schemes that allocate memory corresponding to a r= eturned

=C2=A0 =C2=A0cookie.=C2=A0 Such allocation can be avoided if the server bas= es cookie

=C2=A0 =C2=A0values on a value such as the offset within the directory wher= e the

=C2=A0 =C2=A0scan is to be resumed.


=C2=A0 =C2=A0Cookies generated by such techniques should be designed to rem= ain

=C2=A0 =C2=A0valid despite modification of the associated directory.=C2=A0 = If a server

=C2=A0 =C2=A0were to invalidate a cookie because of a directory modificatio= n,

=C2=A0 =C2=A0READDIRs of large directories might never finish.


So in order to make this work the client would basically have to create its= own B-tree and persist it in storage somewhere.


I don't see the need to make this persistent.=C2=A0 If the client resta= rts, all directory streams have ceased to exist and we know=C2=A0 a posteri= ori=C2=A0 that there are no outstanding directory cookies to which the clie= nt would have to respond.
<mailto:nf= sv4@ietf.org>

--

--

Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com<mailto:trond.myklebust@prima= rydata.com>



--00000000000089a94a05a9873313--