Re: [nfsv4] I-D Action: draft-ietf-nfsv4-rfc5667bis-04.txt

[David Noveck <[e-mail]>]

> I wonder if we can continue to label
> this Direct Data Placement, since the buffer has to be
> "flipped" into place after receive

Good observation :-), although somewhat annoying :-(.

In any case  rtrext-02 will refer to Send-based Data Placement.

On Fri, Feb 3, 2017 at 4:09 PM, Chuck Lever <[e-mail]> wrote:

>
> > On Feb 3, 2017, at 1:04 PM, David Noveck <[e-mail]> wrote:
> >
> > A lot of this reads like pieces of a review of rtrext.  I'll deal with
> some issues within that later but first I want to focus on the concept of
> DDP-eligibility.
> >
> > I'm not sure exactly what it would mean for a concept such as this to be
> completely general and I don't think that's the issue.  For me, the
> important point is that it is a useful concept and that it has more than
> one possible implementation.  XDR items are categorized as either
> DDP-eligible or not.  I am not saying that additional useful
> categorizations cannot arise in the future but I don't know of any and that
> there doesn't seem to be any useful way to prepare for them.
> >
> > The existing categorization is such that there will be individual
> instances of items that implementations will not transfer using DDP.  This
> has always been treated as an implementation choice and ULBs do not say,
> for example, that READs of a few bytes are not DDP-eligible.  Instead, they
> are DDP-eligible and requesters have a way to specify that they be returned
> inline.  To do otherwise would make ULBs overly complicated and very hard
> (perhaps Sisyphean) to maintain.  ULBs are limited to specifying which
> items are DDP-eligible because that is what the requester and responder
> need to agree on.  If the requester wants to send data inline or have
> DDP-eligible response data returned inline that is up to him and ULBs do
> not need to say anything about this choice.
> >
> > The same applies when there are multiple possible forms of DDP with
> different performance characteristics.  The requester chooses and the ULB
> does not need to instruct him about which choice to make.  To try to create
> rules about what choices the requester could make would make a ULB just
> about impossible to do.  Let's not go there.
> >
> > Now let me give some background about rtrext.  Perhaps I've been unduly
> influenced by Talpey's figure of a million 8K random IOs per second, but
> the fact is that I was focused on that case.  In my experience, it is a
> very common case, deserving of significant attention.  I added message
> continuation to rtrext to allow reads of multiples of the block size (e.g.
> 8K) to get the same sort of benefit.
>
> There is no denying that is an important goal.
>
> Unfortunately I've not found a way to approach that level
> of throughput by shrinking per-op latency alone. It's
> important to realize that there is a lower bound for per-op
> latency that is determined by the physics of moving packets
> on a physical network fabric.
>
> High IOPS throughput is actually achieved by using multiple
> QPs through multiple physical interfaces and networks linking
> an SMB client and several storage targets (or in NFS terms:
> pNFS).
>
> So, if one RPC-over-RDMA connection can achieve 125KIOPS (and
> I don't think that's unattainable, even with RPC-over-RDMA V1),
> then eight connections might be able to swing a million IOPS,
> given a careful client implementation. Getting to four
> 250KIOPS connections might be a challenge.
>
> There are some critical pre-requisites to multi-channel
> operation:
>
> - bidirectional RPC on RPC-over-RDMA to enable NFSv4.1
> - multi-path NFS (Andy's work), which works best with NFSv4.1
> - pNFS/RDMA, which requires NFSv4.1, to enable the use of
> multiple DSes
>
> Linux is now close to having all of this in place.
>
>
> > Perhaps I did not devote the attention I should have to IO's that are
> not an integral number of blocks but my experience is that clients are
> typically structured to make this an unusual case.  YMMV.
>
> > I don't claim that send-based DDP is universally applicable and I expect
> other forms of DDP to continue to exist and be useful.  With regard to the
> READ/WRITE distinction, I accept that most of the benefits are for WRITE
> and that many implentations might reasonably choose to use V1-like DDP for
> READ while use send-based DDP for WRITE.  However, there are many
> environments in which remote invalidation is not possible (e.g. user-level
> server) and for these ther are significant benefits from using send-based
> DDP for READ as well.
> >
> > With regard to the sections that read like pieces of a review of rtrext,
> it is hard to respond because the text often makes reference to potential
> extensions for which there is no clear definition.   I think we have to
> address these issues:
> >       • For push mode, it makes sense to define this as another optional
> extension to Version Two.  We need a volunteer to write rpcrdma-pushext.
> If we don't have one now, we should try again after rpcrdma-version-two
> becomes a working group document.
>
> At the moment, push mode looks like a new pNFS layout type,
> thus it is independent of which version of RPC-over-RDMA is
> in use. Agreed, that all has to be written up. Christoph or
> I could make a start at it, but I'm not volunteering anyone's
> time. It would be a short document similar to
>
> http://xml2rfc.tools.ietf.org/public/rfc/bibxml3/reference.
> I-D.draft-hellwig-nfsv4-scsi-layout-nvme-00.xml
>
> which unfortunately appears to have expired.
>
>
> >       • With regard to speculation about a possible form of form of
> Send-based DDP with just Message Continuation, I'm skeptical but if there
> were to be some clear definition, it could be added as an option to
> rtrext-0x, for x>= 2.
>
> I was thinking of starting with an independent document
> (and extension) just to draw the lines clearly. I've been
> busy with other things, and haven't found an opportunity
> to write it up.
>
>
> > With regard to issues related to challenges due to large page sizes, I
> see the matter differently.  I think it is possible to avoid copying
> without using page flipping,but using what I'll call "buffer flipping",
> even when page sizes or large or there is no use of virtual memory at all.
> One might use page flipping only as a way to avoid copies on page-size
> units that are smaller than the buffer size (e.g. 4K pages with 8K buffers).
> >
> > Let's assume the buffer cache uses an 8K buffer size and that the page
> size is either larger or non-existent (i.e.. no virtual memory).  Receive
> buffer are structured to include a section for inline payload (1-2K)
> together with an 8K buffer within the buffer cache.  When posted for
> receive, this buffer should have no valid data within it and it the
> associated buffer header should indicate that is neither free nor assigned
> to a particular file location.  Once data is placed into it by a receive
> which gets a WRITE request, that buffer can be assigned to a location
> within a file and the buffer header updated to reflect this location.  The
> mapping from file and block number to buffer address is changed but the
> address of the buffer remains the same: it is the one into which the buffer
> was placed.
>
> In operating systems I'm familiar with, the page cache
> is the single base of all file I/O. Sub-page sized I/O
> (ie, a separate buffer cache) is no longer utilized.
>
> Even if a particular file system implementation were
> to use it, those buffers would not be exposed above
> the VFS, and thus NFS servers would have no visibility
> of them. Again, that's what I'm familiar with, maybe
> not universally true, and perhaps I've misunderstood
> something.
>
> A Message Continuation approach could use a "flipping"
> mechanism without the requester having to parcel out
> chunk lists.
>
> The key is separating the Transport header from the
> Payload. By fixing the size of the Transport header,
> a receiver can get the incoming Payload to land in
> a separate buffer from the Transport header, always.
>
> A sender might then rely on the fact that the
> receiver's inline threshold is related to it's
> preferred page/buffer cache size, and break each
> message up so that certain data items in the Payload
> stream land in independent buffers on the receiver
> (again, this is similar to an experimental approach
> being considered for NFS on TCP).
>
> RDMA Receive preserves the sender's Send ordering.
> The receiver simply concatenates the incoming messages
> related to the same XID, grouping them as is convenient
> to it.
>
> Receiving such messages should work correctly for any
> arbitrary division of Payload stream contents. Benefits
> accrue when receivers can advertise to senders exactly
> how they want to see RPC payloads divided into messages.
> Perhaps a ULB could be useful here.
>
> This arrangement is still unable to handle what I'll
> term "zero copy sub-buffer modification". In more
> abstract terms, I wonder if we can continue to label
> this Direct Data Placement, since the buffer has to be
> "flipped" into place after receive; it's not a bona
> fide update-in-place. What we are attempting to do
> here is more akin to zero-copy receive.
>
>
> > I think the conformance issues you have mentioned are worth thinking
> about but there will be differences of opinion about how important these
> are.  In any case, the discussion has resulted in some conclusions about
> use of various form of DDP that need to be captured somewhere other than in
> an email thread.
> >
> > I can see why you don't want this stuff in rfc5667bis.  I don't think it
> belongs in nfsulb (or in an update to rfc5667bis based on it) either.  For
> now, I'll add an implementation-choice section to rtrext-02, but ultimately
> I don't think this kind of stuff belongs in a standards-track document.  I
> think that eventually, when the set of extensions in Version Two is more
> settled (and larger than one), we could publish an Informational RFC that
> provides implementation guidance about performance-relevant choices that
> implementers might need to be made aware of.
>
> Yes, that is one possibility.
>
>
> > :
> >
> > On Thu, Feb 2, 2017 at 3:45 PM, Chuck Lever <[e-mail]>
> wrote:
> >
> > > On Feb 2, 2017, at 8:53 AM, David Noveck <[e-mail]> wrote:
> > >
> > > > This section also mentioned inline thresholds and Reply chunks,
> > > > and that looks superfluous. I removed it.
> > >
> > > A whole lot of rfc5667 wound up essentially repeating stuff
> > > in rfc5666.  A lot of that has been removed and that work is
> > > continuing.
> > >
> > > The interesting question is when that process should stop.  In
> > > particular, we need to understand if it can stop short of simply
> > > providing the information required of ULBs in rfc5666bis.
> >
> > I'm totally OK with removing redundant language from rfc5667bis,
> > but we should not pretend that DDP eligibility is completely
> > general or that we understand perfectly how all DDP transport
> > mechanisms will work in the future.
> >
> > I'd like to be sure we are not creating a Sisyphean task.
> >
> >
> > > When
> > > I suggested going that way, you didn't like the idea and I had
> > > doubts about whether it was workable,
> >
> > For example, some repetition of rfc5666bis language is helpful as
> > implementation guidance.
> >
> >
> > > Nevertheless, if we
> > > have material beyond what we need to meet those requirements,
> > > we should understand why it is there.
> >
> > There are other things that can go into a ULB that might not
> > have been listed in rfc5666bis. Clearly one issue we have with
> > NFS in particular is how to deal with replay caching above the
> > transport.
> >
> >
> > > Meanwhile, in nfsulb, I've been going in the opposite direction,
> > > by leaving in the V1-specific material in and indicating that other
> > > implementations are possible and will be discussed in the documents
> > > for the appropriate versions and extensions.  As a result, while
> > > rfc5666bis is getting smaller, my document is larger than -04.
> > >
> > > These documents need to come together at some time, most likely
> > > after rfc5667bis is published (or at least in WGLC) and
> > > rpcrdma-version-two has been turned into a working group
> > > document.
> > >
> > > The prospect of doing so brings us back to our deferred discussion
> > > of the phyletics of the various forms of DDP.  My original plans had
> > > been to put this off until nfsulb was published (done), there was a
> > > new version of rtrext that addressed ULB issues (now working on it),
> and
> > > I added some ULB-related material to rpcrdma-version-two (on my todo
> > > list).
> > >
> > > As I looked at your comments about this issue, I still disagreed with
> them
> > > but decided that they raise issues that the discussion of ULB issues
> > > will have to address, and they might also lead to clarifications in the
> > > rest of rtrext.  So that we don't have to wait for some of these
> documents
> > > to come out (and read them all), let me address your issues below:
> > >
> > > > Explicit RDMA and send-based DDP are perhaps in the same
> > > > phylum but are truly distinct beasts.
> > >
> > > I agree they are different things.  There would be no point in defining
> > > send-based DDP if was the same as V1-DDP.  Nevertheless, they
> > > serve the same purpose: transferring DDP-eligible items so that the
> > > receiver gets them in an appropriately sized and aligned buffer.  I
> > > thinks that is all that matters for this discussion.
> >
> > The numbered items in my previous e-mail demonstrate that the
> > purpose of DDP eligibility is to encourage ULB and implementation
> > choices that are appropriate when using offloaded RDMA. They do not
> > all apply to Send-based DDP, though some do.
> >
> > Or put another way, DDP eligibility is not general at all! It is
> > only "general" because Send-based DDP has been adjusted to fit it.
> >
> > For example, you have argued below that Send-based DDP is aimed at
> > improving a particular size I/O; yet data item size is one important
> > criterion for a ULB author to decide whether to make a data item
> > DDP-eligible.
> >
> > Suppose a ULP has a 512-byte opaque data item (opaque, meaning that
> > no XDR encoding or decoding is required) that would benefit from
> > avoiding a data copy because it is in a frequently-used operation.
> > You might not choose to make it DDP-eligible because an explicit
> > RDMA operation is inefficient. But it might be a suitable candidate
> > for Send-based DDP.
> >
> >
> > > To get back to phyletics, I would put the branching much lower in the
> > > tree.  To me a better analogy is to consider them as two distinct,
> > > closely-related orders, like rodentia and lagomorpha (rabbits,
> > > hares, pikas).
> > >
> > > It is true that there are reasons on might choose to move a
> DDP-eligible
> > > data Item using one or the other but that is an implementation
> decision.
> > > Efficiency considerations  and  direction of transfer may affect this
> > > implementation decision, as is the fact , mentioned in rtrext, that if
> the
> > > client wants the response to be placed at a particular address, you
> > > shoulnd't use send-based DDP.  But none of this affects the fact that
> > > only DDP-eligible items can be placed by send-based DDP, just as is
> the case
> > > with V1-DDP.
> > >
> > >
> > > > 1. The rfc5666bis definition of DDP eligibility makes sense
> > > > if you are dealing with a mechanism that is more efficient
> > > > with large payloads than small. Send-based DDP is just the
> > > > opposite.
> > >
> > > It depends on what you mean by "large" and "small".
> > > Clearly however, neither technique makes sense for very
> > > small payloads and both mske most sense for payloads
> > > over 4K although the specfic will depend on efficiency and
> > > how the send-based DDP receiver chooses to structure its
> > > buffers.
> > >
> > > For me the sweet spot for sen-based DDP would be 8K
> > > transfers and for this reason structured buffers of 9k
> > > (consisting  of 1K primarily for inline payload
> > > together with an 8K aligned segment for direct placement)
> > > are a good choice.   Even larger buffer are possible but the
> > > fact that message continuation can be used means that
> > > send-based DDP should have no problems with 64K
> > > transfers.
> >
> > With Remote Invalidation, NFS READ can be handled with a Write
> > chunk with nearly the same efficiency and latency as a mechanism
> > that uses only Send. Yes, even with small NFS READ payloads,
> > whose performance doesn't matter much anyway. Let's take that off
> > the table.
> >
> > And I think we agree that payloads larger than a few dozen KB are
> > better transferred using explicit RDMA, for a variety of reasons.
> >
> > IMO the only significant case where Send-based DDP is interesting
> > is replacing RDMA Read of moderate-sized payloads (either NFS
> > WRITE data, or Long Call messages). Let's look at NFS WRITE first.
> >
> > Using a large inline threshold (say, 12KB), the server would have
> > to copy some or all of an 8KB payload into place after a large
> > Receive. Server implementations typically already have to copy
> > this data.
> >
> > Why?
> >
> > Because page-flipping a Receive buffer cannot work when the NFS
> > WRITE is modifying data in a file page that already has data on it.
> >
> > Remember that Receive buffers are all the same, all RPC Call
> > traffic goes through them. The hardware picks the Receive buffer
> > arbitrarily; the ULP can't choose in advance which RPC goes into
> > which Receive buffer. Flipping the buffer is the only choice to
> > avoid a data copy.
> >
> > Suppose the server platform's page size is 64KB, or even 2MB.
> > Then, 8KB NFS WRITEs are all sub-page modifications, and copying
> > would be required for each I/O. Explicit RDMA works fine in that
> > case without a data copy.
> >
> > Host CPU data copy has comparable or better latency to flipping
> > a receive buffer into a file, allocating a fresh receive buffer,
> > and DMA mapping it, even for payloads as large as 4KB or 8KB.
> > Eliminating a server-side data copy on small NFS WRITEs is gravy,
> > not real improvement.
> >
> > As far as I can tell, Send-based DDP is not able to handle the
> > sub-page case at all without a copy. That could add an extra rule
> > for using Send-based DDP over and above DDP eligibility, couldn't
> > it?
> >
> > As soon as one realizes that Send-based DDP is effective only
> > when the payload is aligned to the server's page size, depends on
> > the surrounding file contents, and that it is less efficient than
> > explicit RDMA when the payload size is larger than a few KB,
> > introducing the complexity of Send-based DDP to the protocol or
> > an implementation becomes somewhat less appealing.
> >
> > A better approach would attempt to address the actual problem,
> > which is the need to use RDMA Read for moderate payloads. This is
> > for both NFS WRITE and large RPC Call messages.
> >
> > 1. Long Call: large inline threshold or Message Continuation can
> > easily address this case; and it is actually not frequent enough
> > to be a significant performance issue.
> >
> > 2. NFS WRITE: Message Continuation works here, with care taken to
> > align the individual sub-messages to server pages, if that's
> > possible. Or, use push mode so the client uses RDMA Write in this
> > case.
> >
> > The latter case would again be similar to DDP-eligibility, but not
> > the same. Same argument data items are allowed, but additional
> > conditions and treatment are required.
> >
> >
> > > it is true that very large transfers might be better using
> > > V1-DDP but that is beside the point.  Any READ buffer is
> > > DDP-eligible (even for a 1-byte READ) and the choice of
> > > whether to use DDP i(or the sort to use) in any particular
> > > case is up to the implementation.  The ULB shouldn't
> > > be affected.
> >
> > > > 2. Re-assembling an RPC message that has been split by
> > > > message continuation or send-based DDP is done below the XDR
> > > > layer, since all message data is always available once Receive
> > > > is complete, and XDR decoding is completely ordered with the
> > > > receipt of the RPC message. With explicit RDMA, receipt of
> > > > some arguments MAY be deferred until after XDR decoding
> > > > begins.
> > >
> > > First of all, message continuation isn't relevant to this
> > > discussion.
> > >
> > > With regard to send-based DDP, it is true that you have
> > > the placed data immediately, while you might have to wait
> > > in the V1-DDP case, but this doesn't affect  DDP-eligibility.
> >
> > > In either case, before you process the request or response,
> > > all the data, whether inline or placed needs to be available.
> > > I don't see how anything else matters.
> >
> > It can matter greatly for NFS WRITE / RDMA Read.
> >
> > In order for a server to set up the RDMA Read so the payload goes
> > into either a file's page cache or into the correct NVM pages,
> > the server has to parse the NFS header information that indicates
> > what filehandle is involved and the offset and count, before it
> > constructs the RDMA Read(s).
> >
> > Without that information, the server performs the RDMA Read using
> > anonymous pages and either flips those pages into the file, or in
> > the case of NVM, it must copy the data into persistent storage
> > pages associated with that file.
> >
> > That's the reason delayed RDMA Read is still important.
> >
> > Now, Send-based DDP can handle NVM-backed files in one of two ways:
> >
> > 1. Receive into anonymous DRAM pages and copy the payload into NVM;
> > results in a (possibly undesirable) host CPU data copy
> >
> > 2. Receive into NVM Receive buffers; but then all of your NFS
> > operations will have to go through NVM. Maybe not an issue for
> > DRAM backed with flash, but it is a potential problem for
> > technologies like 3D XPoint which are 2x or more slower than
> > DRAM.
> >
> > Remember, for RDMA Read, we are talking about NFS WRITE or
> > pulling moderate-sized arguments from the client on a Call. In
> > that case, a server-side host CPU copy is less likely to interfere
> > with an application workload, and thus is not as onerous as it
> > might be on the client.
> >
> >
> > > > In fact, the current RPC-over-RDMA protocol is designed
> > > > around that. For example the mechanism of reduction relies on
> > > > only whole data items being excised to ensure XDR round-up of
> > > > variable-length data types is correct. That is because an
> > > > implementation MAY perform RDMA Read in the XDR layer, not
> > > > in the transport.
> > >
> > > It may be that send-based DDP could have been designed
> > > to accommodate cases that V1-DDP could not but the fact is that
> > > it wasn't because:
> > >       • There are no common cases in which it would provide a benefit.
> > >       • It would complicate the design to make this something other
> than a drop-in replacement for V1-DDP>
> > > > 3. Message continuation, like a Long message, enables an RPC
> > > > message payload to be split arbitrarily, without any regard
> > > > to DDP eligibility or data item boundaries.
> > >
> > > Exactly right but this just demonstrates that message
> > > continuation and DDP-eligibility should not affect one
> > > another and they don't.
> >
> > Seems to me you can do a form of Send-based DDP with just
> > Message Continuation; perhaps just enough that an additional
> > Send-based DDP mechanism is not necessary.
> >
> > And Send-based DDP isn't terribly valuable without a fully
> > send-based mechanism for transferring large payloads. So
> > it can't be done without Message Continuation, unless I'm
> > mistaken.
> >
> > These two mechanisms seem well-entwined.
> >
> >
> > > > 4. Send-based DDP is not an offload transfer. It involves the
> > > > host CPUs on both ends.
> > >
> > > Not sure what you mean  by that.
> >
> > For Send/Receive, the receiver's host CPU always takes an
> > interrupt, and the host CPU performs transport header
> > processing. (even worse for Message Continuation, where one
> > RPC can now mean more than one interrupt on the receiving
> > CPU).
> >
> > With RDMA Write, for example, the receiver does not take an
> > interrupt, and the wire header is processed by the HCA.
> >
> >
> > >  Both are the same in that:
> > >       • You don't need to copy data on ether side
> > >       • Both sides CPU"s need to deal with where the data is.  The
> adapter has to be told where the data is to sent/transferred from while the
> receiving side has to note where the placed data is (i.e not in the normal
> XDR stream).
> > > > Thus it doesn't make sense to restrict
> > > > the DDP eligibility of data items that require XDR marshaling
> > > > and unmarshaling.
> > >
> > > This sound like you think coping is required.  It isn't.
> >
> > Yes, it is. As above, sometimes with Receive a partial or
> > whole copy of the payload is unavoidable.
> >
> >
> > > There is no point in using send-based DDP to transfer an
> > > item that requires marshalling/unmarshalling and the situation
> > > is the same as  for V1-DDP.  There is no point in specifically
> > > placing data that would get no benefit from being specifically
> > > placed.
> >
> > > > A receiver has generic receive resources that have to
> > > > accommodate any RPC that is received (via Send/Receive).
> > >
> > > True.
> > >
> > > > DDP
> > > > eligibility is designed to prepare the receiver for exactly
> > > > one particular data item that MAY involve the ULP in the
> > > > reconstruction of the whole RPC message.
> > >
> > > True.
> >
> > > On Wed, Feb 1, 2017 at 6:04 PM, Chuck Lever <[e-mail]>
> wrote:
> > >
> > > > On Feb 1, 2017, at 5:04 PM, David Noveck <[e-mail]>
> wrote:
> > > >
> > > > > Within NFS version 4.0, there are a few variable-length result data
> > > > > items whose maximum size cannot be estimated by clients reliably
> > > > > because there is no protocol-specified size limit on these
> > > > > structures.
> > > >
> > > > I like this but I have a problem with the word "structures".   The
> only case
> > > > we are talking about is a GETATTR response, and the XDR defines that
> as an
> > > > opaque array, even though the client and server both are supposed to
> know
> > > > what is in it.  "arrays" would work.
> > >
> > > Agreed, fixed.
> > >
> > > This section also mentioned inline thresholds and Reply chunks,
> > > and that looks superfluous. I removed it.
> > >
> > >
> > > > On Wed, Feb 1, 2017 at 4:47 PM, Chuck Lever <[e-mail]>
> wrote:
> > > >
> > > > > On Feb 1, 2017, at 4:27 PM, karen deitke <[e-mail]>
> wrote:
> > > > >
> > > > >
> > > > >
> > > > > On 1/30/2017 5:28 PM, Chuck Lever wrote:
> > > > >> Hi Karen, thanks for taking time to review.
> > > > >>
> > > > >>
> > > > >>
> > > > >>> On Jan 27, 2017, at 6:40 PM, karen deitke <
> [e-mail]>
> > > > >>>  wrote:
> > > > >>>
> > > > >>>
> > > > >>> Hi Chuck,
> > > > >>>
> > > > >>> o Section 2.6, second to last paragraph, typo, priori should be
> prior.
> > > > >>>
> > > > >> "a priori" is Latin for "in advance" (literally, "from the
> earlier"). I've
> > > > >> replaced this phrase.
> > > > >>
> > > > > Thanks, I don't speak Latin :-)
> > > > >
> > > > >>
> > > > >>
> > > > >>
> > > > >>> o Same section, "Note however that many operations normally
> considered non-idempotent (e.g. WRITE, SETATTR) are actually idempotent".
> This was confusing because prior in the paragraph you     are talking about
> determine the reply size for variable length replies, which doesn't seem to
> be the case for WRITE, SETATTR.
> > > > >>>
> > > > >> That's correct, WRITE doesn't have a variable length reply. This
> section
> > > > >> discusses a possible COMPOUND with WRITE (fixed length result,
> non-idempotent)
> > > > >> and GETATTR (variable length result, idempotent). I've clarified
> it.
> > > > >>
> > > > > ok
> > > > >>
> > > > >>
> > > > >>
> > > > >>> o 4.2.1 seoond paragrah would read better if you removed the
> word "however"
> > > > >>>
> > > > >> I was not able to find the word "however" in the second paragraph
> of S4.2.1.
> > > > >>
> > > > >> There are certain NFS version 4 data items whose size cannot be
> > > > >>
> > > > >> chunk to disambiguate which chunk is associated with which
> argument
> > > > >> estimated by clients reliably, however, because there is no
> protocol-
> > > > >>
> > > > >> data item. However NFS version 4 server and client implementations
> > > > >> specified size limit on these structures. These include:
> > > > > Sorry, it in 4.2:
> > > > >
> > > > > "There are certain NFS version 4 data items whose size cannot be
> estimated by clients reliably, however, because there is no
> protocol-specified size limit on these structures."
> > > >
> > > > How about this:
> > > >
> > > > Within NFS version 4.0, there are a few variable-length result data
> > > > items whose maximum size cannot be estimated by clients reliably
> > > > because there is no protocol-specified size limit on these
> > > > structures.
> > > >
> > > >
> > > > >>>
> > > > >>> "If a client implementation is equipped to recognize that a
> transport error could mean that it provisioned an inadequately sized Reply
> chunk, it can retry the operation with a larger Reply chunk.  Otherwise,
> the client must terminate the RPC transaction."
> > > > >>>
> > > > >>> So you are saying for example that if the client sends a request
> in which it knows that it has "guessed" the maximum response but that guess
> may be wrong, if it detects the connection dropped for example, it could
> potentially interpret that as the guess was too small and send again?
> > > > >>>
> > > > >> As previously discussed.
> > > > >>
> > > > >>
> > > > >>
> > > > >>> 4.3, second from last bullet, "If an READ", should be "If a READ"
> > > > >>>
> > > > >> Fixed.
> > > > >>
> > > > >>
> > > > >>
> > > > >>> 4.5.  Why does the xid in the rdma header have to be used to get
> the session info, wouldnt the session info in the sequence_op that was sent
> in the first place have this info?
> > > > >>>
> > > > >> Here's the last paragraph of Section 4.5:
> > > > >>
> > > > >>    In addition, within the error response, the requester does not
> have
> > > > >>    the result of the execution of the SEQUENCE operation, which
> > > > >>    identifies the session, slot, and sequence id for the request
> which
> > > > >>    has failed.  The xid associated with the request, obtained
> from the
> > > > >>    rdma_xid field of the RDMA_ERROR or RDMA_MSG message, must be
> used to
> > > > >>    determine the session and slot for the request which failed,
> and the
> > > > >>    slot must be properly retired.  If this is not done, the slot
> could
> > > > >>    be rendered permanently unavailable.
> > > > >>
> > > > >> The mention of RDMA_MSG is probably incorrect. In that case, the
> server
> > > > >> was able to return a SEQUENCE result, and that should be useable
> by the
> > > > >> client.
> > > > >>
> > > > >> Without an RPC Reply message, however, the client matches the XID
> in the
> > > > >> ERR_CHUNK message to a previous call and that will have the
> matching
> > > > >> SEQUENCE operation.
> > > > >>
> > > > > I'm still not following this.  Is the issue that the  client would
> resend, but the seq of the slot is incremented and we wouldn't potentially
> get anything the server had in the slot replay?
> > > >
> > > > Is there a way to recover if the server can't send a SEQUENCE result
> for
> > > > that slot? Seems like the slot would be stuck on the same RPC until
> the
> > > > session is destroyed.
> > > >
> > > >
> > > > > Karen
> > > > >>
> > > > >> That makes this rather a layering violation, and perhaps a reason
> why
> > > > >> retransmitting with a larger Reply chunk might be a cure worse
> than the
> > > > >> disease.
> > > > >>
> > > > >> I'm beginning to believe that making this situation always a
> permanent
> > > > >> error, as rfc5666bis does, is a better protocol choice.
> > > > >>
> > > > >>
> > > > >>
> > > > >>> Karen
> > > > >>> On 1/20/2017 5:27 PM,
> > > > >>> [e-mail]
> > > > >>>  wrote:
> > > > >>>
> > > > >>>> A New Internet-Draft is available from the on-line
> Internet-Drafts directories.
> > > > >>>> This draft is a work item of the Network File System Version 4
> of the IETF.
> > > > >>>>
> > > > >>>>         Title           : Network File System (NFS) Upper Layer
> Binding To RPC-Over-RDMA
> > > > >>>>         Author          : Charles Lever
> > > > >>>>    Filename        : draft-ietf-nfsv4-rfc5667bis-04.txt
> > > > >>>>    Pages           : 18
> > > > >>>>    Date            : 2017-01-20
> > > > >>>>
> > > > >>>> Abstract:
> > > > >>>>    This document specifies Upper Layer Bindings of Network File
> System
> > > > >>>>    (NFS) protocol versions to RPC-over-RDMA.  Upper Layer
> Bindings are
> > > > >>>>    required to enable RPC-based protocols, such as NFS, to use
> Direct
> > > > >>>>    Data Placement on RPC-over-RDMA.  This document obsoletes
> RFC 5667.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> The IETF datatracker status page for this draft is:
> > > > >>>>
> > > > >>>>
> > > > >>>> https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc5667bis/
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> There's also a htmlized version available at:
> > > > >>>>
> > > > >>>>
> > > > >>>> https://tools.ietf.org/html/draft-ietf-nfsv4-rfc5667bis-04
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> A diff from the previous version is available at:
> > > > >>>>
> > > > >>>>
> > > > >>>> https://www.ietf.org/rfcdiff?url2=draft-ietf-nfsv4-
> rfc5667bis-04
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> Please note that it may take a couple of minutes from the time
> of submission
> > > > >>>> until the htmlized version and diff are available at
> tools.ietf.org.
> > > > >>>>
> > > > >>>> Internet-Drafts are also available by anonymous FTP at:
> > > > >>>>
> > > > >>>>
> > > > >>>> ftp://ftp.ietf.org/internet-drafts/
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> _______________________________________________
> > > > >>>> nfsv4 mailing list
> > > > >>>>
> > > > >>>>
> > > > >>>> [e-mail]
> > > > >>>> https://www.ietf.org/mailman/listinfo/nfsv4
> > > > >>> _______________________________________________
> > > > >>> nfsv4 mailing list
> > > > >>>
> > > > >>> [e-mail]
> > > > >>> https://www.ietf.org/mailman/listinfo/nfsv4
> > > > >> --
> > > > >> Chuck Lever
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >
> > > >
> > > > --
> > > > Chuck Lever
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > nfsv4 mailing list
> > > > [e-mail]
> > > > https://www.ietf.org/mailman/listinfo/nfsv4
> > > >
> > >
> > > --
> > > Chuck Lever
> > >
> > >
> > >
> > >
> >
> > --
> > Chuck Lever
> >
> >
> >
> >
>
> --
> Chuck Lever
>
>
>
>