Re: [nfsv4] rfc5667bis open issues

[David Noveck <[e-mail]>]

> > As far as the session case, the server will consider the request
executed but the client does not have a reply containing the slot and
sequence.  To deal with that case, he would have to use rdma_xid in the
message with the ERR_CHUNK to get the needed context and so conclude that
the slot was available for reuse.

> This feels like something that belongs in 5667bis. I'm no real
> expert on session behavior. Can you sketch some text that can > be added?

I'll write it assuming it will constitute a new subsection 4.x dealing with
session-related issues.

On Mon, Sep 26, 2016 at 1:20 PM, Chuck Lever <[e-mail]> wrote:

>
> > On Sep 24, 2016, at 8:11 AM, David Noveck <[e-mail]> wrote:
> >
> > >The issue is that the language allows ERR_CHUNK to be
> > > returned after a server has processed the RPC request when
> > > a client has not provided adequate Write list or Reply chunk
> > > resources to convey the reply.
> >
> > I forgot that we made ERR_CHUNK ambiguous, apparently because of
> reluctance to add things to the XDR.  There is no pressing need to do this
> since the responder is aware of the difference and his management of the
> DRC can follow based on his knowledge.
> >
> > However, in the context of Version Two it would be better if we avoided
> those ambiguities, given that we have lots of space for distinct error
> codes.
> >
> > > In that case, it makes sense for the server to have added
> > > the request to its DRC.
> >
> > Agree that in the case in which the server has executed the request, it
> should add the request to the DRC.  In practical terms, there are not
> likely to be cases in which there is a non-idempotent request with a reply
> longer than 1K.
>
> For NFSv3, that is largely true.
>
> For NFSv4, I believe a large reply to a non-idempotent request is
> possible, and may even be common. Any time a client does something
> like this:
>
>   { SEQUENCE, PUTFH, SETATTR, GETATTR }
>
> Where the GETATTR requests an ACL or security label, is problematic
> if the client does not estimate the reply buffer size correctly.
>
> However, this case is for when the client has a bug; it's not a
> case where we expect one or the other side to perform heroic
> recovery. ERR_CHUNK would terminate the RPC on the client, which
> would very likely return EIO to the application. I think that's
> about the sanest outcome we can expect.
>
>
> > As far as the session case, the server will consider the request
> executed but the client does not have a reply containing the slot and
> sequence.  To deal with that case, he would have to use rdma_xid in the
> message with the ERR_CHUNK to get the needed context and so conclude that
> the slot was available for reuse.
>
> This feels like something that belongs in 5667bis. I'm no real
> expert on session behavior. Can you sketch some text that can be
> added?
>
>
> > > What I propose is that if the first READ_PLUS returns
> > > NFS4_CONTENT_HOLE, the server would return an empty first
> > > Write chunk. Then the second READ_PLUS result always
> > > lines up with the second Write chunk, which IMO is much better > for
> clients.
> >
> > I'm OK with this but I think you will need to adjust the text to reflect
> the fact that READ_PLUS can return an array of read_plus_content's,
> although, in practice, those that return more than one are extremely rare.
>
> I hadn't realized READ_PLUS returned an array.
>
> If an NFS server is allowed to structure its reply in a way that
> the client cannot predict, then I think we'll have to limit the
> way READ_PLUS uses DDP. I propose these rules:
>
> - The client can provide no more than one Write chunk if it expects
> NFS4_CONTENT_DATA. (No Write chunk or an empty Write chunk, following
> the previous rules, would be for when the client predicts that the
> reply can go inline).
>
> - If that Write chunk is non-empty, it MUST be large enough to
> receive all expected payload bytes in a single NFS4_CONTENT_DATA
> element.
>
> - The server uses that Write chunk for the first array element that
> has an NFS4_CONTENT_DATA arm.
>
>
> Then we have a choice, depending on whether it is more desirable
> to return data in a single round-trip, or more desirable to preserve
> holes. Either:
>
> - If the server finds that the array has grown larger than can be
> returned inline or via the supplied Reply chunk, it MUST return
> the payload data in a single NFS4_CONTENT_DATA element via the
> provided Write chunk.
>
> Or:
>
> - The server MUST return as much payload as it can fit within the
> resources provided by the client, and return it as a short READ
> result. The client is responsible for retrying the READ_PLUS to
> read the remaining payload.
>
> Somehow we have to deal with the case where the server cannot fit
> any of the payload in the client-provided resources.
>
>
> READ_PLUS is actually a poor fit for offloaded DDP anyway. The whole
> point of offload is that the client has to do no work; the payload
> arrives in its memory without any effort on its part.
>
> I would just as soon require that, on RDMA transports, READ_PLUS
> returns only a hole or exactly one contiguous piece of content.
>
>
> --
> Chuck Lever
>
>
>
>