[nfsv4] Notes regarding discussion of directory scalabiliy issues

[David Noveck <[e-mail]>]

*Introduction*

On 6/22, Chuck and I held a discussion to try to resolve some issues that
arose in our discussion of scalability issues for directory operations.
 After our original presentations on this topic at the
post-IETF107 virtual interim, it was anticipated that we would discuss
those on the wg mailing list.   Since that didn't work out, Chuck and I
decided to clarify and possibly resolve our differences of approach in a
short meeting that worked well in the form of a phone call.

We were able to clarify but not resolve two issues.  We hope to be able to
resolve these eventually, but not necessarily during the July 9th meeting.

   - We did agree that we will need to better understand, and try to
   resolve issues regarding the compatibility of directory notifications and
   client handling of directory cookies and their caching. See *Directory
   Delegation Issues* for details.


   - We also explored issues raised by the possible addition of ops to aid
   recursive directory traversals, building on Chuck's suggestion, at the
   earlier meeting, of possible protocol aid for "rm -r". See  *Ops to Add
   Aiding Recursive Directory Traversal *for details

*Directory Delegation Issues*

Implementations of directory delegations are quite limited, making it not
worth investing in server-side implementations.   Useful client-side
implementation would require the ability to efficiently cache large slowly
changing directories.   Unfortunately, that is not currently possible, at
least for the Linux client, since the creation or removal of a file from a
directory requires that the entire directory (which can be very large) be
refetched.  This now occurs for directory changes that the client makes
itself but would presumably also apply in the case of directory
notifications, making them not very useful in the important case of large
slowly-changing directories.

As Chuck related to me, this requirement for directory refetching is
predicated on the possibility that any change to a directory (e.g. remove,
create, rename) could potentially invalidate directory entry cookies for
all the cached directory.entries.   While this is unlikely to happen in
practice, I can see that clients might be unwilling to rely on filsystems
not changing these. The thing I don't understand, and Chuck was unable to
clarify for me, is why such entries need to be cached at all, and
essentially treated as if they were attributes.  While the directory
notification feature has the ability to propagate attribute changes to
clients with delegations, there is no such ability in the case of cookie
changes.  It appears that the feature was designed assuming these would not
be necessary.

As a result, it appeared that one if the following would have to be done:

   1. Clients might be modified so as not to depend on a supposed fixity of
   such cookies. That's clearly my favorite as I believe the client returning
   a cached directory could synthesize its own cookies to allow users to fetch
   directory information across multiple requests.  However, I'm not sure
   client implementers would agree and this is a matter on which consensus is
   important.
   2. Provide a way in which the server could  communicate that the
   theoretical possibility that a remove, create, or rename could cause a
   revision of cookies for uninvolved directory entries does not occur for  a
   given fs. This could be done by adding a new fs-scope attribute providing
   information about an fs's directory entry cookie management.  The downside
   is that this would be v4.2-only and would be require an additional RFC to
   make a v4.1 feature effectively usable 😞
   3. Provide another way to exclude the possibility of the client being
   blind-sided by a server-side fs prone to directory cookie reassignment.
   For example, any change to a directory entry cookie not being removed or
   renamed could require delegation recall.  Unfortunately, this is a big
   change to an existing feature😖

As Chuck and I finished the discussion we anticipated a long-term process
aimed at securing a consensus about which of these choices the working
group would adopt to make directory delegations useful.  This could start
at the 7/9 meeting but the need to make sure we had the
active participation of client implementers meant it wasn't a sure thing
for initial discussion at the 7/9 meeting.

Lately, I've been looking at directory notifications in more detail and
come to the conclusion that the way the notifications use cookies to update
cached directories is really predicated on the expectation that cookies for
directory entries not involved in the specific update will not change.  For
example, insert and rename notifications include the cookie of the entry
before the insertion point, which is not really useful if the server-side
fs  is free to change cookies for entries not involved in the directory
operation.

I now believe that it is possible to rework the description of this feature
so that the server by supporting directory notifications is providing
assurance that the server, by supporting such notifications, is effectively
providing an assurance that the wholesale revision of cookies as a result
of directory modification operations, about which clients are concerned,
cannot happen.   This could not be done as a consequence of an errata
report, but it is the sort of clarification/revision that I feel could be
done as a part of rfc5661bis, assuming that we can reach a working group
consensus on the matter.  We intend to do change sof this scale for some
REJECTED errata reports.I will take about 5-10 minutes for a presentation
of this issue at the 7/9 meeting, hoping to stimulate later discussion of
this issue and future directory delegation/notification
implementation possibilities on the mailing list.

*Ops to Add Aiding Recursive Directory Traversal*

In Chuck's presentation he alluded to the possibility of the protocol
giving "rm -r" more help and there are a number of useful extensions that
could be defined.  As I considered other applications in which recursive
directory traversal there appeared to be a number of possible READDIR
extensions that could be sensibly proposed and would be useful in software
build workloads.

The problem with such extensions is that they are unlikely to be used
unless complementary work is done to provide useful API's to provide access
to any helpful protocol extensions.  Although such work is out-of-scope for
the working group, the working group has to avoid investing in protocol
extensions which, realistically, will never be used.  As a result I will
not present regarding possible extensions at the 7/9 meeting but may dp
this later if there is interest in compatible API's for important clients.

*Miscellaneous Items.*

We also had occasion to discuss some other issues regarding the agenda of
the  forthcoming meeting:

   - Chuck reminded me of my previously mentioned intention to use github
   for the handling of the writing and review of wg documents associated with
   rfc5661bis (i.e. draft-ietf-nfsv4-internationalization,
   draft-ietf-nfsv4-security-needs, draft-ietf-nfsv4-security,
   draft-ietf-nfsv4-rfc5661bis, and possibly others).   Chuck wasn't clear
   what exactly I might need from him to help the process and it turned out,
   I wasn't sure either.  We agreed that I would mention the issues in my
   general slides on the rfc5661bis process and that I'd give Chuck an early
   opportunity to respond to those.
   - Chuck pointed out the issue of the length of rfc5661 nd the need to
   address concerns about that.  I was concerned that previous suggestions in
   this regard (with regard to pNFS file) might result in multiple documents
   that don't fit together all that well.   I agreed to look at other
   approaches to the issue and present those to the working group at the 7/9
   meeting.