Re: [nfsv4] Review of draft-ietf-nfsv4-rfc5667bis-09

[David Noveck <[e-mail]>]

> The RFC2119 term RECOMMENDED is basically a synonym for SHOULD. It's
> perfectly permissible.

Not if it's implementation advice.

On Thu, Apr 27, 2017 at 11:54 AM, Tom Talpey <[e-mail]> wrote:

> On 4/27/2017 11:44 AM, Chuck Lever wrote:
>
>>
>> On Apr 27, 2017, at 7:20 AM, Tom Talpey <[e-mail]> wrote:
>>>
>>> snip:
>>>>    Such implementation limits can constrain the complexity of NFS
>>>>    version 4 COMPOUNDs, limit the number of elements in scatter-gather
>>>>    operations, or prevent the use of Kerberos integrity or privacy
>>>>    services.
>>>>
>>>
>>> I like the approach, and the lead-in language looks good. The text
>>> quoted above is just a little bit dark, especially that bit about
>>> preventing krb5i/krb5p. I'd suggest a more active statement to replace
>>> the above, including the more prescriptive SHOULD rather than "can".
>>> How about:
>>>
>>> "Client implementations SHOULD be prepared to provide mechanisms for
>>>  reporting the above errors, and optionally provide configuration to
>>>  limit the complexity of NFS version 4 COMPOUNDs, limit the number
>>>  of elements in scatter-gather operations, and to avoid other possible
>>>  sources of RPC-over-RDMA chunk overruns at the peer.
>>>
>>>  These become especially important when Kerberos integrity or privacy
>>>  is in place for the RPC connection. These facilities add payload to
>>>  the RPC headers, potentially increasing the complexity of the chunk
>>>  manipulation, independent of the upper layer NFS operation. The
>>>  implementation SHOULD consider such RPC payload requirements in
>>>  addition to the NFS considerations."
>>>
>>
>> Sure, I can work this in.
>>
>> When you say "Client implementations SHOULD ... [report] the above
>> errors" you are talking about reporting them to administrators
>> and/or RPC consumers? I don't think we can use SHOULD in that case.
>>
>
> I am agnostic about who to inform. The important thing is that some
> visibility of the error be surfaced. I absolutely don't think an
> arbitrary GARBAGE_ARGS returned to an application that may simply
> choke on it, qualifies.
>
> This feels like implementation advice, not protocol.
>>
>
> Correct. But since the protocol creates the problem, the protocol
> definition needs to say something about dealing with it. So I believe
> SHOULD is best.
>
> Would "recommend" be enough for this section?
>>
>
> The RFC2119 term RECOMMENDED is basically a synonym for SHOULD. It's
> perfectly permissible.
>
> Tom.
>
>
>
>>
>> Feel free to wordsmith further.
>>>
>>> Tom.
>>>
>>>
>>> On 4/26/2017 12:18 PM, Chuck Lever wrote:
>>>
>>>>
>>>> On Apr 24, 2017, at 7:30 AM, Chuck Lever <[e-mail]>
>>>>> wrote:
>>>>>
>>>>>
>>>>>> On Apr 24, 2017, at 7:56 AM, Tom Talpey <[e-mail]> wrote:
>>>>>>
>>>>>> On 4/21/2017 10:43 AM, Chuck Lever wrote:
>>>>>>
>>>>>>> I agree that SHOULD/MAY makes things cloudier, and does not
>>>>>>> seem to align with well-defined RFC2119 usage.
>>>>>>>
>>>>>>> Another way we've dealt with similar disagreements between
>>>>>>> specification and implementation is to decide that one of
>>>>>>> the implementations is incorrect.
>>>>>>>
>>>>>>> Can we agree that:
>>>>>>>
>>>>>>> - GARBAGE_ARGS is a bit of a layering violation, though it's
>>>>>>> understandable why it might be returned
>>>>>>>
>>>>>>> - RPC clients are already prepared for GARBAGE_ARGS
>>>>>>>
>>>>>>
>>>>>> Are you certain of this?
>>>>>>
>>>>>
>>>>> GARBAGE_ARGS has been part of the RPC protocol for decades.
>>>>> The two Unix-flavored clients that have NFS/RDMA support can
>>>>> both handle this error.
>>>>>
>>>>
>>>> I've confirmed that the only other known NFS/RDMA client
>>>> (Oracle dNFS) properly recognizes GARBAGE_ARGS.
>>>>
>>>>
>>>> And out of curiosity, what is returned
>>>>>> to the consumer for GARBAGE_ARGS versus ERR_CHUNK?
>>>>>>
>>>>>
>>>>> RFC 5531:
>>>>>
>>>>>> GARBAGE_ARGS  = 4, /* procedure can’t decode params   */
>>>>>>
>>>>>
>>>>>
>>>>> GARBAGE_ARGS is an RPC-level error. The reply is "accepted"
>>>>> with accept_stat GARBAGE_ARGS. An XID is available in the
>>>>> header.
>>>>>
>>>>> rfc5666bis:
>>>>>
>>>>>> If the rdma_vers field contains a recognized value, but an
>>>>>> XDR parsing error occurs, the responder MUST reply with an
>>>>>> RDMA_ERROR procedure and set the rdma_err value to ERR_CHUNK.
>>>>>>
>>>>>
>>>>>
>>>>> ERR_CHUNK is a transport level error. An XID is available
>>>>> in the header.
>>>>>
>>>>> The difference is that the RPC layer v. the transport layer
>>>>> are reporting they don't understand the contents of the
>>>>> message (Call). There is nothing more in either type of
>>>>> message.
>>>>>
>>>>>
>>>>> - In RPC-over-RDMA Version One, we are not trying to recover
>>>>>>> (in the sense of resending a simpler COMPOUND) but are rather
>>>>>>> trying to ensure the offending RPC is properly terminated on
>>>>>>> the client, and does not further block other RPCs or deadlock
>>>>>>> the transport
>>>>>>>
>>>>>>> Thus I claim it is harmless if a server returns GARBAGE_ARGS
>>>>>>> instead of ERR_CHUNK.
>>>>>>>
>>>>>>
>>>>>> "Harmless" is a bit relative. The operation fails, through no fault
>>>>>> of the consumer. And, frankly, in a very mysterious way.
>>>>>>
>>>>>
>>>>> We have no richer way of communicating failure in RPC-over-RDMA
>>>>> Version One. We are not looking for recovery here, so I don't
>>>>> believe any more information would be useful. If the server
>>>>> wishes, it can log the failure with a message explaining what
>>>>> went wrong.
>>>>>
>>>>>
>>>>> Again, I think there is more to say here. It's a limitation of the
>>>>>> protocol whose implications should be made clear (contraining the
>>>>>> complexity of COMPOUNDs, limiting scatter/gather lengths, etc).
>>>>>>
>>>>>
>>>>> I'd welcome any suggested text.
>>>>>
>>>>> Honestly, I'm not sure what can be said. Neither NFSv4.0 nor
>>>>> RPC-over-RDMA have a sophisticated mechanism to communicate this
>>>>> kind of limitation. The best an NFSv4 server can do is return
>>>>> NFS4ERR_RESOURCE, which also carries little extra information
>>>>> about what a client should do to recover.
>>>>>
>>>>> So are you comfortable with eliminating GARBAGE_ARGS if we can
>>>>> come up with more detail about the impact of not knowing how
>>>>> complex a COMPOUND can be?
>>>>>
>>>>
>>>> I've come up with some possible replacement text for
>>>> the final two paragraphs of S5.4.1 in an attempt to
>>>> address comments from Tom, David, and Karen. The
>>>> normative requirements have been removed, and a (brief)
>>>> discussion of the consequences of not handling complex
>>>> COMPOUNDs was introduced.
>>>>
>>>>
>>>> 5.4.2.  Complexity Considerations
>>>>
>>>>   As mentioned above, an NFS version 4 COMPOUND procedure can contain
>>>>   more than one operation that carries a DDP-eligible data item.  The
>>>>   RPC-over-RDMA Version One protocol does not place any limit on the
>>>>   number of chunks that may appear in the Read or Write lists.
>>>>   Therefore an NFS version 4 client MAY construct an RPC-over-RDMA
>>>>   Version One message containing more than one Read chunk or Write
>>>>   chunk.
>>>>
>>>>   However, implementations have practical limits on the number of
>>>>   chunks or segments they are prepared to process in one of these
>>>>   lists.  There are several ways an NFS Version 4 server might indicate
>>>>   that an RPC Call message constructed by a client is valid but cannot
>>>>   be processed because of implementation limitations:
>>>>
>>>>   o  If the problem is detected in the upper layer (i.e., by the NFS
>>>>      version 4 implementation), the server returns an NFS status of
>>>>      NFS4ERR_RESOURCE.
>>>>
>>>>   o  If the problem is detected during XDR decoding of the request
>>>>      (e.g., during re-assembly of the Call message by the RPC layer),
>>>>      the server returns an RPC accept_stat of GARBAGE_ARGS.
>>>>
>>>>   o  If the problem is detected at the transport layer (i.e., during
>>>>      transport header processing), the server returns an RDMA_ERROR
>>>>      message with the err_code field set to ERR_CHUNK.
>>>>
>>>>   Such implementation limits can constrain the complexity of NFS
>>>>   version 4 COMPOUNDs, limit the number of elements in scatter-gather
>>>>   operations, or prevent the use of Kerberos integrity or privacy
>>>>   services.
>>>>
>>>>
>>>> Comments, opinions on this approach?
>>>>
>>>>
>>>> Tom.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> As a result, I can change the Read list text in S5.4.1 to be
>>>>>>> the same as the Write list text, removing the mention of
>>>>>>> GARBAGE_ARGS.
>>>>>>>
>>>>>>> Would that sit comfortably with everyone?
>>>>>>>
>>>>>>>
>>>>>>> On Apr 20, 2017, at 7:21 PM, David Noveck <[e-mail]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> The "or" is a similar situation, it prescribes a choice, which
>>>>>>>>> does not define a protocol.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Fair enough, but the point that needs to be made is that, with
>>>>>>>> regard to Version One, Chuck and  the working group is not
>>>>>>>> free to define a protocol.  As a result we have the kind of
>>>>>>>> ugliness you object to, but it is inherent in the choice to try to
>>>>>>>> revive Version One as-is.
>>>>>>>>
>>>>>>>> If an NFS version 4 client sends an RPC Call with a Read list that
>>>>>>>>> contains more chunks than an NFS version 4 server is prepared to
>>>>>>>>> process, the server SHOULD reject the request by responding with an
>>>>>>>>> RDMA_ERROR message with the rdma_err value set to ERR_CHUNK. The
>>>>>>>>> server MAY reject the RPC with an RDMA_MSG message containing an
>>>>>>>>> RPC
>>>>>>>>> Reply with an accept status of GARBAGE_ARGS.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I think I know what you intend here and I've seen stuff like this
>>>>>>>> in RFCs but I don't
>>>>>>>> wthink e can do this because this is not in line with the
>>>>>>>> definitions of "SHOULD"
>>>>>>>> and "MAY" that appear in RFC2119.
>>>>>>>>
>>>>>>>> "SHOULD" means that you are supposed to do something but can avoid
>>>>>>>> it if
>>>>>>>> you have a good reason and are aware of the consequences of not
>>>>>>>> doing it.
>>>>>>>> In this case the "good" reason is that someone coded the
>>>>>>>> implementation
>>>>>>>> to do something else, which is not all that good a reason.  The
>>>>>>>> consequences of
>>>>>>>> returning the GARBAGEARGS are exactly zero, since the client has to
>>>>>>>> be prepared
>>>>>>>> for either it or ERR_CHUNK.
>>>>>>>>
>>>>>>>> "MAY" means the implementation can choose to do the action or not,
>>>>>>>> which is line
>>>>>>>> with the reality here but essentially contradicts the SHOULD.
>>>>>>>>
>>>>>>>> This at least makes it clear which response is "preferred".
>>>>>>>>>
>>>>>>>>
>>>>>>>> But it is isn't really the job of the RFC2119 terms to say which is
>>>>>>>> "preferred" or
>>>>>>>> "'preferred'".  These terms are supposed to describe
>>>>>>>> interoperability and the
>>>>>>>> interoperability situation is that the server MUST return ERR_CHUNK
>>>>>>>> or
>>>>>>>> GARBAGEARGS and the client needs to be prepared for either.  That
>>>>>>>> is the
>>>>>>>> unpleasant reality.  If you want to indicate a preference, you can
>>>>>>>> say something
>>>>>>>> like:
>>>>>>>>         • Returning ERR_CHUNK is preferrable.
>>>>>>>>         • Returinng ERR_CHUNK is more in line with the appropriate
>>>>>>>> protocol layering since this issue relates to a limitation of the transport
>>>>>>>> implementation.
>>>>>>>>         • Use of GARBAGEARGS is an unfortunate artifact of
>>>>>>>> inappropriately layered implementations and is only allowed for reasons of
>>>>>>>> compatibility with existing implementations.  It is desirable to avoid it.
>>>>>>>>
>>>>>>>>> And one would hope a future draft would decide.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Not sure what draft you are thinking of.  I don't see us doing an
>>>>>>>> rfc5667bisbis (rfc5667tris).
>>>>>>>>
>>>>>>>> By the time we did that, the implementations with these
>>>>>>>> restrictions will probably be gone.
>>>>>>>>
>>>>>>>> I have a second question though. How does the client determine what
>>>>>>>>> is
>>>>>>>>> the actual error? As in, how many chunks were allowed?
>>>>>>>>>
>>>>>>>>
>>>>>>>> This is not fixable in Version One.  It would be in Version Two,
>>>>>>>> but by then
>>>>>>>> the need will probably be gone.
>>>>>>>>
>>>>>>>> Does the upper
>>>>>>>>> layer have to recover, and if so what are the implications?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I think something could be put in to indicate that clients should
>>>>>>>> break up COMPOUNDS
>>>>>>>> so the only have a single chunk each.
>>>>>>>>
>>>>>>>> Yes, I know 5667 did not explore this very well.
>>>>>>>>>
>>>>>>>>
>>>>>>>> It didn't explore it at all.  And 5666's error reporting facilities
>>>>>>>> were extremely limited.
>>>>>>>>
>>>>>>>> Mea culpa.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I don'tt think you have anything to apologize for.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Apr 20, 2017 at 5:28 PM, Tom Talpey <[e-mail]> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 4/19/2017 11:14 AM, Chuck Lever wrote:
>>>>>>>> Hi Tom-
>>>>>>>>
>>>>>>>> On Apr 18, 2017, at 11:08 PM, Tom Talpey <[e-mail]> wrote:
>>>>>>>>
>>>>>>>> I noticed the same thing, and I'll add that the MUST reject
>>>>>>>> condition
>>>>>>>> is very confusing because it allows an "or". In my opinion a MUST is
>>>>>>>> always a single requirement, never ambiguous.
>>>>>>>>
>>>>>>>> I agree this kind of thing is tricky. I wrote it as "the server MUST
>>>>>>>> reject the RPC". That's the single requirement. The choice is how
>>>>>>>> the
>>>>>>>> rejection is conveyed to the client.
>>>>>>>>
>>>>>>>> The statement "MUST reject" is not testable. So, while it may be
>>>>>>>> understood what is intended, there is nothing implementable in the
>>>>>>>> MUST. The "or" is a similar situation, it prescribes a choice, which
>>>>>>>> does not define a protocol.
>>>>>>>>
>>>>>>>> Is there some reason you want to allow such a choice? I think you'll
>>>>>>>> find that, worded properly, it becomes actually much less
>>>>>>>> implementable
>>>>>>>> and interoperable than you may think.
>>>>>>>>
>>>>>>>> The Solaris server can return an RPC-level error in cases like this.
>>>>>>>>
>>>>>>>> Well, this is happening because the Solaris server is (probably)
>>>>>>>> just
>>>>>>>> handing the chunk list up to the RPC layer, and it's the RPC (XDR)
>>>>>>>> processing that detects any problem.
>>>>>>>>
>>>>>>>> On the other hand, an implementation could do the opposite, it could
>>>>>>>> process the chunks at the lower layer, before ever invoking RPC
>>>>>>>> processing. This would naturally lead to a non-RPC error.
>>>>>>>>
>>>>>>>> The challenge in defining the protocol is to hide these
>>>>>>>> possibilities.
>>>>>>>>
>>>>>>>> I think there are similar choices allowed in rfc5666bis. Let's say
>>>>>>>> that in a perfect world, I would go with only ERR_CHUNK, but I'm
>>>>>>>> documenting existing implementation behavior here.
>>>>>>>>
>>>>>>>> I'm not sure it matters to the client: both errors are permanent and
>>>>>>>> the RPC is terminated on the client.
>>>>>>>>
>>>>>>>> I'm open to alternatives.
>>>>>>>>
>>>>>>>> The icky way to do this is to split into two weak requirements.
>>>>>>>>
>>>>>>>> If an NFS version 4 client sends an RPC Call with a Read list that
>>>>>>>> contains more chunks than an NFS version 4 server is prepared to
>>>>>>>> process, the server SHOULD reject the request by responding with an
>>>>>>>> RDMA_ERROR message with the rdma_err value set to ERR_CHUNK. The
>>>>>>>> server MAY reject the RPC with an RDMA_MSG message containing an RPC
>>>>>>>> Reply with an accept status of GARBAGE_ARGS.
>>>>>>>>
>>>>>>>> This at least makes it clear which response is "preferred". And one
>>>>>>>> would hope a future draft would decide.
>>>>>>>>
>>>>>>>> I have a second question though. How does the client determine what
>>>>>>>> is
>>>>>>>> the actual error? As in, how many chunks were allowed? Does the
>>>>>>>> upper
>>>>>>>> layer have to recover, and if so what are the implications?
>>>>>>>>
>>>>>>>> Yes, I know 5667 did not explore this very well. Mea culpa.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Tom.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 4/18/2017 6:32 PM, karen deitke wrote:
>>>>>>>> Hi Chuck, its unclear what you mean by "is prepared to process" in
>>>>>>>> the text below.
>>>>>>>> Other than that, looks good.
>>>>>>>>
>>>>>>>> Karen
>>>>>>>>
>>>>>>>> 5.4.1
>>>>>>>> If an NFS version 4 client sends an RPC Call with a Write list that
>>>>>>>> contains more chunks than an NFS version 4 server is prepared to
>>>>>>>> process, the server MUST reject the RPC by responding with an
>>>>>>>> RDMA_ERROR message with the rdma_err value set to ERR_CHUNK.
>>>>>>>>
>>>>>>>>
>>>>>>>> If an NFS version 4 client sends an RPC Call with a Read list that
>>>>>>>> contains more chunks than an NFS version 4 server is prepared to
>>>>>>>> process, the server MUST reject the RPC by responding with an
>>>>>>>> RDMA_MSG message containing an RPC Reply with an accept status of
>>>>>>>> GARBAGE_ARGS, or with an RDMA_ERROR message with the rdma_err value
>>>>>>>> set to ERR_CHUNK.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 4/18/2017 1:21 PM, David Noveck wrote:
>>>>>>>> *Overall Evaluation*
>>>>>>>> *
>>>>>>>> *
>>>>>>>> Major improvement over RFC5667.  Almost ready to ship.  No technical
>>>>>>>> issues.
>>>>>>>>
>>>>>>>> A lot of my comments are basically editorial and are offered on a
>>>>>>>> take-it-or-lease-it basis.
>>>>>>>>
>>>>>>>> I think some clarification in Section 5.4.1 is needed although not
>>>>>>>> necessarily in the ways suggested below,
>>>>>>>>
>>>>>>>> *Comments by Section*
>>>>>>>> *5.4.1. Multiple DDP-eligible Data Items*
>>>>>>>> Giventhat READ_PLUS no longer has any DDP-eligible data items, the
>>>>>>>> situation described in the fifth bullet can no longer arise. I
>>>>>>>> suggest
>>>>>>>> deleting the bullet.
>>>>>>>> The penultimate paragraph can be read as applying to some situations
>>>>>>>> in which it shouldn't and where the extra chunks would very
>>>>>>>> naturally
>>>>>>>> ignored. For example, if you had on write chunk together with a READ
>>>>>>>> operation which failed, the server would have more chunks (i.e. one)
>>>>>>>> than the number it is prepared to process (i.e. zero). Suggest, as a
>>>>>>>> possible replacement:
>>>>>>>>
>>>>>>>> Normally, when an NFS version 4 client sends an RPC Call with a
>>>>>>>> Write list that contains multiple chunks. each such, when matched
>>>>>>>> with a DDP-eligible data item in the response, directs the
>>>>>>>> placement of the data item as specified by
>>>>>>>> [I.D.-nfsv4-rfc5666bis]. When there are DDP-eligible data items
>>>>>>>> matched to write chunks that an NFS version 4 server is not
>>>>>>>> prepared to process, the server MUST reject the RPC by responding
>>>>>>>> with an RDMA_ERROR message with the rdma_err value set to ERR_CHUNK.
>>>>>>>>
>>>>>>>> With regard to the last paragraph, I am curious that this paragraph,
>>>>>>>> unlike the previous one, allows GARBGEARGS. Is this so because that
>>>>>>>> would be allowed if the chunks in question had offsets other than
>>>>>>>> those that correspond to DDP-eligible data items? If so, please
>>>>>>>> consider the following possible replacement.
>>>>>>>>
>>>>>>>> Normally, when an NFS version 4 client sends an RPC Call with a
>>>>>>>> Read list that contains multiple chunks, each such, when properly
>>>>>>>> matched with a DDP-eliigible data item in the request, directs the
>>>>>>>> fetching of the the data item as specified by
>>>>>>>> [I.D.-nfsv4-rfc5666bis]. When there are DDP-eligible data items
>>>>>>>> matched to read chunks that an NFS version 4 server is not
>>>>>>>> prepared to process, the server MUST reject the RPC by responding
>>>>>>>> with an RDMA_ERROR message with the rdma_err value set to ERR_CHUNK.
>>>>>>>>
>>>>>>>> *5.6. Session-Related Considerations*
>>>>>>>> In the third sentence of the second paragraph, suggest replacing "no
>>>>>>>> different" by "not different".
>>>>>>>> In the last sentence of the last paragraph, suggest replacing "is
>>>>>>>> not"
>>>>>>>> by "were not"
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> nfsv4 mailing list
>>>>>>>> [e-mail]
>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> nfsv4 mailing list
>>>>>>>> [e-mail]
>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> nfsv4 mailing list
>>>>>>>> [e-mail]
>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>>
>>>>>>>> --
>>>>>>>> Chuck Lever
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> nfsv4 mailing list
>>>>>>>> [e-mail]
>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> nfsv4 mailing list
>>>>>>>> [e-mail]
>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> nfsv4 mailing list
>>>>>>>> [e-mail]
>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Chuck Lever
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> nfsv4 mailing list
>>>>>>> [e-mail]
>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> nfsv4 mailing list
>>>>>> [e-mail]
>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>
>>>>>
>>>>> --
>>>>> Chuck Lever
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> nfsv4 mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>
>>>>
>>>> --
>>>> Chuck Lever
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> nfsv4 mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>
>>>>
>>> _______________________________________________
>>> nfsv4 mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>
>>
>> --
>> Chuck Lever
>>
>>
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/nfsv4
>>
>>
> _______________________________________________
> nfsv4 mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/nfsv4
>