[nfsv4] Changes to rfc5661sesqui suggested by errata 2006.

[David Noveck <[e-mail]>]

I've been looking at the changes it makes sense to make to rfc5611sesqui to
adapt to the changes suggested by errata 2006.  You can find the
actual changed text below but first let me summarize the changes.

The largest change consists of a rewritten section 15.1,1.3.  This include
the changes suggested by errata 2006 but also has the following additional
changes:

   - It adds to the list of possible reasons to return NFS4ERR_DELAY an
   example (session migration described in section 11.13.3) since none of the
   existing examples covers the case of returning NFS4ERR_DELAY on a SEQUENCE
   operation.
   - It deals with the possible effect of NFS4ERR_DELAY on replies fetched
   from the replay cache, including issues related to non-idempotent requests.
   - It explicitly deals with requests that do not include a SEQUENCE
   operation.


In addition, there are some limited changes:

   - An additional bullet item is added to the list of things that stiil
   need to be done in rfc5661bis (in Section 1.1), covering the erratas other
   than 2006.
   - A reference to section 15.1.1.3 is added to Section 11.12.2.
   - A number of references to Section 15.1.1.3 are added to Section
   11.12.3.
   - A description of the changes made to Section 15.1.1.3 is added to
   Appendix B.3.

If people are OK with these changes, i can submit a srfc5661sesqui-nsns-02
whenever it is
appropriate.

*Replaced Section 15.1.1.3 <http://15.1.1.3>:*

15.1.1.3.  NFS4ERR_DELAY (Error Code 10008)

   For any of a number of reasons, the replier could not process this
   operation in what was deemed a reasonable time.  The client should
   wait and then try the request with a new slot and sequence value.

   Some examples of scenarios that might lead to this situation:

   o  A server that supports hierarchical storage receives a request to
      process a file that had been migrated.

   o  An operation requires a delegation recall to proceed, so that the
      need to wait for for this delegation to be recalled makes
      processing this request in a timely fashion impossible.

   o  A request is being performed on a session being migrated from
      another server as described in Section 11.13.3, and the lack of
      full information about the state of the session on the source
      makes it impossible to process the request immediately

   In such cases, returning the error NFS4ERR_DELAY allows necessary
   preparatory operations to proceed without holding up requester
   resources such as a session slot.  After delaying for period of time,
   the client can then re-send the operation in question, often as part
   of a nearly identical request.  Because of the need to avoid spurious
   reissues of non-idempotent operations and to avoid acting in response
   to NFS4ERR_DELAY errors returned on responses returned from the
   replier's replay cache, integration with the session-provided replay
   cache is necessary.  There are a number of cases to deal with, each
   of which requires different sorts of handling by the requester and
   replier:

   o  If NFS4ERR_DELAY is returned on a SEQUENCE operation, the request
      is retried in full with the SEQUENCE operation containing the same
      slot and sequece values.  In this case, the replier MUST avoid
      returning a response containing NFS4ERR_DELAY as the response to
      SEQUENCE solely on the basis of its presence in the replay cache.
      If the replier did this, the retries would not be effective as
      there would be no opportunity for the replier to see whether the
      condition that generated the NFS4ERR_DELAY had been rectified
      during the interim between the original request and the retry.

   o  If NFS4ERR_DELAY is returned on a operation other than SEQUENCE
      which validly appears as the first operatoion of a request,
      handling is similar.  The request can be retired in full without
      modification.  In this case as well, the replier MUST avoid
      returning a response containing NFS4ERR_DELAY as the response to
      an intial operation of a request solely on the basis of its
      presence in the replay cache.  If the replier did this, the
      retries would not be effective as there would be no opportunity
      for the replier to see whether condition that generated the
      NFS4ERR_DELAY had been rectified during the interim between the
      original request and the retry.

   o  If NFS4ERR_DELAY is returned on an operation other than the first
      in the request, the request when retried MUST contain a SEQUENCE
      operation which is different than the original one, with either
      the bin id or the sequence value different from that in the
      original request.  Because requesters do this, there is no need
      for the replier to take special care to avoid returing an
      NFS4ERR_DELAY error, obtained from the replay cache.  When no non-
      idempotent operations have been processed before the NFS4ERR_DELAY
      was returned, the requester should retry the request in full, with
      the only difference from the original request being the
      modfication to the slot id or sequence value in the reissued
      SEQUENCE operation.

   o  When NFS4ERR_DELAY is returned on an operation other than the
      first within a request and there has been a non-idempotent
      operation processed before the NFS4ERR_DELAY was returned, the
      reissued request should avoid the non-idempotent operation.  The
      request still must use a SEQUENCE operation with either a
      different slot id or sequence value from the SEQUENCE in the
      original request.  Because this is done, there is no way the
      replier could avoid spuriously re-eecuting the non-idempotent
      operation since the different SEQUENCE parameters prevent the
      requester from recognizing that the non-idempotent operation is
      being retried.

   Note that without the ability to return NFS4ERR_DELAY and the
   requester's willingness to re-send when receiving it, deadlock might
   result.  For example, if a recall is done, and if the delegation
   return or operations preparatory to delegation return are held up by
   other operations that need the delegation to be returned, session
   slots might not be available.  The result could be deadlock.

*Addition to Section 1.1:*

The following is to be added as the third bullet in the list in this
section:

   o  Work would have to be done to address many erratas relevant to RFC
      5661, other than errata 2006 which is addressed in this document.
      That errata was not deferrable because of the interaction of the
      changes suggested in that errata and handling of state and session
      migration.  The erratas that have been deferred include changes
      originally suggested by a p[articular errata, which change
      consensus decisions made in RFC5661, which need to be changed to
      ensure compatibility with existing implementations that do not
      follow the handling delineated in RFC 5661.  Note that it is
      expected that such erratas will remain relevant to implementers
      and the authors of an eventual rfc5661bis, despite the fact that
      this document, when approved, will obsolete RFC 5661

*Modifications in Section 11.12.2:*

The fifth non-bulleted paragraph is modified to add a new last sentence and
now reads as follows:

      When transferring state between the source and destination, the
   issues discussed in Section 7.2 of [63] must still be attended to.
   In this case, the use of NFS4ERR_DELAY may still necessary in
   NFSv4.1, as it was in NFSv4.0, to prevent locking state changing
   while it is being transferred.  See Section 15.1.1.3 for information
   about appropriate client retry approaches inn the event that
   NFS4ERR_DELAY is returned.

*Modifications in Section 11.12.3:*

The sixth non-bulleted paragraph is modified to add a new last sentence and
now reads as follows:

      An important issue is that the specification needs to take note of
   all potential COMPOUNDs, even if they might be unlikely in practice.
   For example, a COMPOUND is allowed to access multiple file systems
   and might perform non-idempotent operations in some of them before
   accessing a file system being migrated.  Also, a COMPOUND may return
   considerable data in the response, before being rejected with
      NFS4ERR_DELAY or NFS4ERR_MOVED, and may in addition be marked as
   sa_cachethis.  However, note that if the client and server adhere to
   rules in Section 15.1.1.3, there is no possibility of non-idempotent
   operations being spuriouly reissued after receiving NFS4ERR_DELAY
   response.

The seventh non-bulleted paragraph is modified to read as it appear below.
Added
material is italicized:

   Because of the considerations mentioned above
*including the rules for   the handling of NFS4ERR_DELAY included in
Section 15.1.1.3*, the
   destination server can respond appropriately to SEQUENCE operations
   received from the client by adopting the three policies listed below:

*Addition to Appendex B.3:*

The following new bulleted paragraph needs to be added at the end of
Appendix B.3:

   o  Because of the need to provide the clarifications in errata 2006
      and to adapt these to properly explain the interaction of
      NFS4ERR_DELAY with the replay cache, a revised description of
      NFS4ERR_DELAY appears in Section 15.1.1.3.  This errata, unlike
      manay other RFC5661 erratas is addressed in this document because
      of the extensive use of NFS4ERR_DELAY in connection with state
      migration and session migration.