Re: [nfsv4] RPC/RDMA read chunk round-up when inline content follows

[Chuck Lever <[e-mail]>]

Hi Dave-

On Jan 7, 2015, at 4:58 AM, David Noveck <[e-mail]> wrote:

> 
> ---------- Forwarded message ----------
> From: David Noveck <[e-mail]>
> Date: Tue, Jan 6, 2015 at 2:56 PM
> Subject: Re: [nfsv4] RPC/RDMA read chunk round-up when inline content follows
> To: "J. Bruce Fields" <[e-mail]>
> 
> 
> > So if I understand correctly: fixing the servers will make them fail all
> > writes from current clients,
> 
> If it hurts when you do that, don't do that.  it seems you can make servers
> accept old and new client behavior, even if the old behavior is a BAD IDEA.

Whether or not existing implementations support the old way, new
servers should never have to. Arguably the spec already does not
allow it, thus this point is moot: existing implementations are
broken, according to RFC 5667, and always have been. The question
for the WG is whether the spec should be updated to explicitly
grandfather this behavior.

Implementers are free to do what is needed to march over the
incompatibility, including choosing temporarily to be non-
compliant with the specification (eg. to allow both ways during
a period of transition). No spec change is needed to do that.

A practical question is whether allowing servers to continue
accepting the old way is at all risky (that is, would affect an
implementation’s ability to interoperate in the future). I think
that would be valuable information for implementers who are faced
with the decision, and would also inform the WG about whether a
spec change is advisable.

Both the Solaris and Linux servers would need to be fixed to
accept the correct way. Then at a later time, they would need
to be updated to reject the old way. That seems feasible.

The only client implementation at issue is the Linux client. The
Solaris client sends only { PUTFH, WRITE }. Another way to put it
is a spec change is being requested to accommodate one broken
client implementation. That seems awkward.


> one way to map this into RFC2119 language is to say the servers SHOULD reject
> older-form requests and MUST accept newer-form requests.
> 
> 
> > andf fixing the clients will make the writes fail against all current servers.
> 
> not sure how to state this in rfc2119 language but I'd argue that one should not fix the
> clients until severs are fixed to accept newer-form request.
> 
> 
> > Isn't it a little late for that kind of change?  
> 
> I think it is.

IMO it is not too late because practically speaking, this is not
the only interop issue that is facing these incomplete
implementations, whether or not they are deployed by customers.

Linux, for instance, does not support RDMA_NOMSG type RPC calls on
either the client or server. RDMA_NOMSG type calls are needed for
SYMLINK and large NFSv4 COMPOUNDs. Would the WG consider removing
RDMA_NOMSG from the spec, or do we rightly consider this a necessary
but missing feature in Linux?

At this stage I don’t think anyone has tried using a Solaris
client with a Linux server. I’m guessing there will be interop
problems there too.


> > Or are there really so
> > few NFS/RDMA users that we can afford to completely break backwards
> > compatibility?
> 
> i guess it depends on whether you are one of those users.

This affects only NFSv4 on RDMA, and there is a work-around:
use NFS on IPoIB until the client and server interoperate again.


> The approach described in draft-ietf-nfsv4-versioning is to make server support for old-form
> and new-form requests into separate features that the client can test for, allowing clients
> to be fixed to adapt to server behavior in parallel with making servers which support both 
> behaviors.
> 
> Not only is that quite heavyweight for this problem, but this draft is a ways away from 
> IESG approval and publishing.
> 
> 
> On Tue, Jan 6, 2015 at 1:58 PM, J. Bruce Fields <[e-mail]> wrote:
> On Mon, Jan 05, 2015 at 06:50:44PM +0000, Tom Talpey wrote:
> > I hope this additional reply doesn't make the thread too convoluted.
> >
> > > -----Original Message-----
> > > From: nfsv4 [mailto:[e-mail]] On Behalf Of Chuck Lever
> > > Sent: Thursday, January 1, 2015 1:51 PM
> > > To: David Noveck
> > > Cc: Chunli Zhang; nfsv4 list ([e-mail]); Karen; Dai Ngo
> > > Subject: Re: [nfsv4] RPC/RDMA read chunk round-up when inline content
> > > follows
> > >
> > > Hi Dave-
> > >
> > > Sorry for the length. There's a lot to tease apart.
> > >
> > >
> > > On Jan 1, 2015, at 7:20 AM, David Noveck <[e-mail]> wrote:
> > >
> > > > > It turns out that Linux is the only NFS client that supports RDMA
> > > > > and adds an operation (GETATTR) after WRITE in an NFSv4 compound.
> > > >
> > > > Presumably it sends the GETATTR as a separate chunk.  I suppose that
> > > > could be a performance issue.
> > >
> > > Or efficiency: extra work has to be done to register that 16 byte piece of the
> > > compound RPC and send it in a separate RDMA READ. That's extra transport
> > > bytes on the wire, and an extra round trip (albeit a very fast one).
> >
> > It's my opinion this is a bug, in both the client and the server here.
> >
> > The intention of an RDMA Chunk is that it encodes all or part of an RPC
> > argument, RPC result, or an entire RPC message. The fact that the client
> > added a compounded GETATTR to the write data's Read Chunk is wrong.
> > The fact that the server parsed the additional data after the write payload
> > as an RPC request is also wrong. Keeping these segments separate is critical
> > to the RFC5667 specification of the NFS binding to RPC/RDMA.
> 
> So if I understand correctly: fixing the servers will make them fail all
> writes from current clients, and fixing the clients will make their
> writes fail against all current servers.
> 
> Isn't it a little late for that kind of change?  Or are there really so
> few NFS/RDMA users that we can afford to completely break backwards
> compatibility?
> 
> --b.
> 
> >
> > That said, the text in RFC5667 may be unclear. I'd suggest that it needs to
> > be revisited anyway, to reflect implementation experience as well as the
> > new behaviors of NFSv4.1, v4.2 etc. The document is five years old, and it
> > reflects work done prior to that. In particular, while it was published at the
> > same time as RFC5661 (NFSv4.1), it actually doesn't make any specific
> > requirements for that minor version beyond mentioning the callback
> > channel. That, plus pNFS, layouts, CREATE, OPENATTR, ACLs, etc.
> >
> > >
> > > > > Linux sends the part of the compound before the opaque file data via
> > > > > RDMA SEND. The opaque file data is put into the read list. Then the
> > > > > GETATTR operation is put in the read list after the file data.
> > > >
> > > > > Existing NFS/RDMA servers are able to receive and process this
> > > > > request correctly.
> > > >
> > > > The interesting question is whether they would work if one switched
> > > > from the chunks-only-at-the-end approach to the alternative
> > > > chunks-only-for-WRITE approach.
> > >
> > > The current Linux NFS server and the Solaris 11 update 2 NFS server do not
> > > handle a request with additional inline content. There is a minor exception,
> > > but let's put that aside for the moment.
> > >
> > > Recall that the Linux NFS client sends { PUTFH, WRITE, GETATTR } as its
> > > NFSv4 WRITE compound. I've constructed a Linux client that sends the
> > > GETATTR as additional inline content rather than at the end of the read list.
> > >
> > > The Linux server accepts the PUTFH and WRITE, but then returns 10044
> > > (OP_ILLEGAL) for the third op, because it finds nonsense when it looks in the
> > > XDR stream for the arguments of the third op. (I do have a fix for this).
> > >
> > > I can make the converse point: sending an RPC request where only a middle
> > > argument is placed in a chunk list is clearly allowed by RFC 5666, and is not
> > > forbidden by RFC 5667. Having "position" fields for each chunk in the read list
> > > means a server MUST be prepared to accept inline content following a read
> > > list
> >
> > I'm not sure I agree with "MUST", but I do agree that there is sufficient
> > (redundant) information in the encoding of the request for the server to
> > Perform such an action.
> >
> > >
> > > Therefore current NFS/RDMA server implementations are broken. If we
> > > agree that sending the final GETATTR in the read list is allowed, then servers
> > > should accept that in addition to NFSv4 compounds where the final GETATTR
> > > is sent as additional inline content.
> > >
> > > > > First question: does the Linux client comply with RFCs 5666 and 5667
> > > > > when it sends an NFSv4 WRITE compound in this way, or should it be
> > > > > changed?
> > > >
> > > > It clearly complies with RFC5666 which grants the sender a lot of
> > > > freedom for the sender as to how it chooses to sends individual data
> > > elements.
> >
> > While freedom is the intention of RFC5666, it is (was) the intention of RFC5667
> > to narrowly define those freedoms for the NFS protocol family. There is a
> > requirement in RFC5666 that each upper layer provide such a binding, with
> > the appropriate rules to ensure interoperability.
> >
> > > >
> > > > > The intention of RFC 5667 appears to be that the GETATTR belongs
> > > > > inline, following the front of the compound.
> > > >
> > > > I think the basic idea is that RDMA is for large data elements, such
> > > > as data to be written, but I don't think it comes out and forbids you
> > > > from sending other things.
> > >
> > > This text from section 4 seems to disallow the use of a read list for anything
> > > but opaque file data or symlink pathnames:
> > >
> > > "Similarly, a single RDMA Read list entry MAY be posted by the client  to
> > > supply the opaque file data for a WRITE request or the pathname  for a
> > > SYMLINK request.  The server MUST ignore any Read list for  other NFS
> > > procedures, as well as additional Read list entries beyond  the first in the list."
> >
> > Note however that RFC5667 section 4 is explicitly about NFS versions 2 and 3,
> > and the "MUST ignore" statement is purely about simplifying the nfsdirect
> > layering to align with the (simple) requirements of NFS v2 and v3, transfer-wise.
> >
> > Section 5 in turn is silent on the issue of SYMLINK, which of course is implemented
> > In NFSv4.x as the CREATE procedure. That's clearly an omission.
> >
> > I agree with many of the points you make below, but in the absence of clear
> > normative statements in RFC5667, I think they simply reinforce the need
> > for updating the nfsdirect specification.
> >
> > Tom.
> >
> >
> >
> > >
> > > "The server MUST ignore" is a clear statement that placing other arguments
> > > in a read list will prevent interoperation, at least for
> > > NFSv2 and NFSv3 Direct.
> > >
> > > The "MAY" refers to an alternate mechanism of sending large RPC
> > > requests: RDMA_NOMSG, where the entire RPC message is encapsulated in
> > > a read list at position 0.
> > >
> > > > The only basis on which you might base a case that this violates
> > > > RFC5667 wiule be based on the following text from section2:
> > > >
> > > > Large chunks of data,
> > > > such as the file data of an NFS WRITE request, MAY be referenced by an
> > > > RDMA Read list and be moved efficiently and directly placed by an RDMA
> > > > Read operation initiated by the server.
> > > >
> > > > One could argue that this somehow implies you "MAY NOT" transfer other
> > > > sorts of request data using RDMA read chunks.  I don't read it that way.
> > >
> > > > > Only the file data
> > > > > payload belongs in the read list, if NFSv4 Direct is to be
> > > > > consistent with NFSv2 and NFSv3.
> > > >
> > > > I don't think NFSv4 can be consistent with previous NFS protocols
> > > > because it is different in having COMPOUND.
> > >
> > > I don't agree that NFSv4 cannot be consistent with legacy NFS. Perhaps it is
> > > not stated clearly in RFC 5667 section 5 how to make NFSv4 equivalent to
> > > NFSv2/3 Direct, but observe that:
> > >
> > > 1. If you consider only WRITE operations, sending only the opaque file
> > >    data payload is allowed in all three cases.
> > >
> > > 2. If you consider NFSv2 SYMLINK, there is an argument (sattr)
> > >    following the link pathname argument. RFC 5667 says only the link
> > >    pathname may be sent in a read list. This is a clear requirement
> > >    that the middle argument is conveyed via a read list, and the
> > >    remaining arguments inline. The case where both the pathname and
> > >    the sattr argument are sent in a read list is explicitly not
> > >    allowed.
> > >
> > > The legacy-consistent way of sending an NFSv4 WRITE compound would be
> > > to put only the opaque file data in a read list, no matter what else appears in
> > > the NFSv4 COMPOUND request with the WRITE operation. (Small WRITEs of
> > > course are always sent inline).
> > >
> > > > Your choice is either to be consistent with NFSv3:
> > > >   * In having chunks only at the end of the request.
> > > >   * In having chunks only for WRITE data.
> > >
> > > RFC 5667 section 5 has two sentences discussing the use of read lists with
> > > NFSv4 COMPOUND requests:
> > >
> > > "The situation is similar for RDMA Read lists sent by the client and  applies to
> > > the NFSv4.0 WRITE and SYMLINK procedures as for v3.
> > >  Additionally, inline segments too large to fit in posted buffers MAY  be
> > > transferred in special "RDMA_NOMSG" messages."
> > >
> > > Although it does suggest strongly that NFSv4 WRITE should be consistent
> > > with NFSv3 WRITE, at first glance it doesn't help us decide which of your two
> > > bullets is the correct interpretation.
> > >
> > > "The situation is similar" I believe references earlier text in section 5, not to
> > > the text in section 4 that discusses WRITE/SYMLINK. Earlier, section 5 goes as
> > > far as to say:
> > >
> > > "The Write list MUST be considered only for the COMPOUND procedure.
> > >  This procedure returns results from a sequence of operations.  Only  the
> > > opaque file data from an NFS READ operation and the pathname from  a
> > > READLINK operation MUST utilize entries from the Write list."
> > >
> > > The "Only . . . MUST" construction in the third sentence is slippery.
> > >
> > > It can be read as allowing other operations and arguments to use a write list,
> > > and as requiring READ and READLINK to use a write list in all cases, including
> > > short payloads. That reading, while literal, seems contradictory with the rest
> > > of the document.
> > >
> > > Or it can be read as allowing a write list only for READ and READLINK
> > > operations in NFSv4 compounds. That reading I believe is consistent with the
> > > rest of the document and earlier versions of the NFS Direct protocol, and
> > > would also apply to WRITE in the read list case.
> > >
> > > Also worth mentioning is that NFSv4 does not have a separate SYMLINK
> > > operation. Thus the explicit mention of an NFSv4 SYMLINK operation in
> > > section 5 is incorrect. It could be replaced with CREATE(NF4LNK). Or using a
> > > read list during symlink creation could simply be abandoned for NFSv4, in
> > > favor of sending large CREATE requests via RDMA_NOMSG.
> > >
> > > In sum, a very particular reading of section 5 suggests that only WRITE
> > > payloads in NFSv4 COMPOUND requests can be sent in a read list, but there
> > > seems to be some wiggle room. It would help me, as an implementer, if RFC
> > > 5667 section 5 could be clarified.
> > >
> > > > > Now suppose the Linux client is changed to place the GETATTR
> > > > > operation inline, following the front of the compound.
> > > >
> > > > And assuming the existing servers are OK with that?
> > >
> > > > > When the opaque file data requires round-up because it's length is
> > > > > not divisible by four, should there be a pad?
> > > >
> > > > No.
> > > >
> > > > > if so, where does it belong?
> > > >
> > > > > To put it another way: Since inline data does not have a "position"
> > > > > field, should the receiver assume that inline content following the
> > > > > read list begins right at the end of the rounded-up chunk?
> > > >
> > > > No.
> > > >
> > > > > or should
> > > > > the receiver assume the inline content begins at the next XDR
> > > > > position after the end of the chunk?
> > > >
> > > > Yes.
> > > >
> > > > > If the former, that would require inserting a zero pad before the
> > > > > inline content. But that pad would cause the following inline
> > > > > content to be sent unaligned, since a zero pad is always shorter
> > > > > than 4 bytes long.
> > > >
> > > > > By implication, then, the receiver MUST conclude that round-up is
> > > > > present in the case when inline data remains to be decoded (ie, the
> > > > > following inline content always begins at the next XDR position).
> > > > > The sender MUST NOT send a pad inline in this case. Is this correct?
> > > >
> > > > Yes.
> > >
> > > Thanks for confirming my interpretation, and for helping me to sharpen my
> > > understanding of these RFCs.
> > >
> > > > > I've read RFC 5666 section 3.7. Paragraph 4 seems most relevant, but
> > > > > it doesn't discuss the case where there is additional inline content
> > > > > following a chunk list.
> > > >
> > > > I think the following is relevant:
> > > >
> > > > Because this position will not match (else roundup would not have
> > > > occurred), the receiver decoding will fall back to inspecting the
> > > > remaining inline portion.
> > > >
> > > > This may not be clear enough.  Maybe a clarifying editorial errata
> > > > would be justified.
> > >
> > > Let me know if you'd like me to file one.
> > >
> > > > On Tue, Dec 30, 2014 at 12:53 PM, Chuck Lever <[e-mail]>
> > > wrote:
> > > > Hi-
> > > >
> > > > It turns out that Linux is the only NFS client that supports RDMA and
> > > > adds an operation (GETATTR) after WRITE in an NFSv4 compound.
> > > >
> > > > Linux sends the part of the compound before the opaque file data via
> > > > RDMA SEND. The opaque file data is put into the read list. Then the
> > > > GETATTR operation is put in the read list after the file data.
> > > >
> > > > Existing NFS/RDMA servers are able to receive and process this request
> > > > correctly.
> > > >
> > > > First question: does the Linux client comply with RFCs 5666 and 5667
> > > > when it sends an NFSv4 WRITE compound in this way, or should it be
> > > > changed?
> > > >
> > > > The intention of RFC 5667 appears to be that the GETATTR belongs
> > > > inline, following the front of the compound. Only the file data
> > > > payload belongs in the read list, if NFSv4 Direct is to be consistent
> > > > with NFSv2 and NFSv3
> > > >
> > > > Now suppose the Linux client is changed to place the GETATTR operation
> > > > inline, following the front of the compound.
> > > >
> > > > When the opaque file data requires round-up because it's length is not
> > > > divisible by four, should there be a pad? if so, where does it belong?
> > > >
> > > > To put it another way: Since inline data does not have a "position"
> > > > field, should the receiver assume that inline content following the
> > > > read list begins right at the end of the rounded-up chunk? or should
> > > > the receiver assume the inline content begins at the next XDR position
> > > > after the end of the chunk?
> > > >
> > > > If the former, that would require inserting a zero pad before the
> > > > inline content. But that pad would cause the following inline content
> > > > to be sent unaligned, since a zero pad is always shorter than 4 bytes
> > > > long.
> > > >
> > > > By implication, then, the receiver MUST conclude that round-up is
> > > > present in the case when inline data remains to be decoded (ie, the
> > > > following inline content always begins at the next XDR position). The
> > > > sender MUST NOT send a pad inline in this case. Is this correct?
> > > >
> > > > I've read RFC 5666 section 3.7. Paragraph 4 seems most relevant, but
> > > > it doesn't discuss the case where there is additional inline content
> > > > following a chunk list.
> > > >
> > > > Thanks for reading!
> > > >
> > > > --
> > > > Chuck Lever
> > > >
> > > > _______________________________________________
> > > > nfsv4 mailing list
> > > > [e-mail]
> > > > https://www.ietf.org/mailman/listinfo/nfsv4
> > > >
> > >
> > > --
> > > Chuck Lever
> > > chuck[dot]lever[at]oracle[dot]com
> > >
> > >
> > >
> > > _______________________________________________
> > > nfsv4 mailing list
> > > [e-mail]
> > > https://www.ietf.org/mailman/listinfo/nfsv4
> >
> > _______________________________________________
> > nfsv4 mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/nfsv4
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/nfsv4

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com