Re: [nfsv4] NFS/RDMA next steps

[David Noveck <[e-mail]>]

> Note that RFC 5667bis is now "Submitted to IESG for Publication". The work
> items in the slides assume that this document will be completed and
published
> as planned.

It's been that way for seventeen days.  I expect it to be done in about six
months but RFC 8154 took about a year.  Sigh!

> The slides split the possibilities into three somewhat orthogonal
groupings.

You had to divide these up to produce slides, but everyone will divide
things
differently.  In any case, I don't see these divisions as all that
helpful.  I think we
need to decide what documents are ready to become working group
documents based on their maturity and promise and believe that is the
appropriate way to frame the discussion.

> Opinions
> are welcome as to what order, whether something is left out, or what
might be
> removed from this list. Groupings Two and Three could introduce new
Working
> Group documents, and thus have implications for our charter milestone
count.

Note that Grouping One does as well.  See below.


> Grouping Zero:
> Focus on improving existing implementations of RPC-over-RDMA and
NFS/RDMA. No
> IETF action needed, which is why I didn't include this on the slides.

For the same reason it might not be all that appropriate to this
discussion.  The balance
between iimplementation and specfication is a decision for individual
companies to
make.  In my case, the work is concerned with the development of a new
implementation
but the same principle applies.  My employer will have opinions about this
question and
I'm going to pay attenntion to my company's needs.  It is not up to me how
Oracle people
spend their time or up to Oracle about how mine is spent.

> There
> are substantial improvements that can be made to existing base
implementations,
> but these would be done by many of the same folks who would be working on
new
> protocol.

True.  If people are so busy doing implementation that they have no time to
work on
specs we will have to adjust.

> Grouping One:
> Enable greater transport parallelism in NFS.

This strikes me as a highly unnatural grouping to use as a basis for
decision-making.
It might make sense in a slide preentation.

> This includes multipathing

I'm working on this.  I mentioned prodcucing
draft-dnoveck-nfsv4-mv1-msns-update at the meeting.

My understanding is that Chuck will submit draft-cel-mv0-trunking-update
which addresses
trunking/multipathing in the v4.0 context.

> and
> use of pNFS with RDMA. No changes to RPC-over-RDMA or NFS/RDMA are
necessary,
> and this would bring important performance capabilities to NFS, especially
> by enabling very low latency client access to Storage Class Memory.

I agree with doing this but what we heard at ietf99 was a very early
draft.  This is a long-term effort which should be pursued but this
document is quite a ways from being ready to become
a working group document.

> Grouping Two:
> Incrementally improve RPC-over-RDMA version 1. The main idea here is to
> introduce a per-connection transport property negotiation mechanism to
replace
> CCP. This would enable variable size (ie larger) inline thresholds and
the use
> of Remote Invalidation in some instances with existing deployments.

Implementation of this is important and will be proceding.  I intend to
implement it
regardlersss of working group decisions about priorities or the pospect of
controversy.
Perhaps this belongs in group Zero.  The work to get this
already completed specfication work through the IETF process is relatively
small although there may be controversies that need to be resolved.  See
below.


> Grouping Three:
> Pursue RPC-over-RDMA version 2. This would open a variety of avenues by
which
> many of the perceived shortcomings of RPC-over-RDMA version 1 could be
> addressed.


> IMO Zero and One are where we can get the greatest bang for the buck in
the
> near term.

Two issues with this:

   - I believe Zero is out of scope for this discussion for reasons already
   given.
   - The pNFS-RDMA layout type is not a near-term item.  I agree we should
   pursue it, but it is not going to happen soon, especially if much of the
   working group is busy doing implementation work.  In any case, when
   Christoph thinks this is ready to be a working group item, we should
   definitely consider this seriously.

> The current proposal for Grouping Two (draft-cel-nfsv4-rpcrdma-cm-pvt-msg)
is
> controversial.

if Tom, or anyone else, has objections to proceeding with this, we need to
discuss this on the list and work to arrive at a reasonable resolution.  My
understaning is that Tom objects to this being standards-track.  While I
believe it should be standards-track, I can live with an Informatinal RFC.
  I just don't want ths to be an undocumented de facto standard.

> Grouping Three would be an immense amount of work to generalize
> some things for less gain than we might see with work in Grouping Zero or
One.

I don't think "immense" is the right word but let's not argue about that.
In any case, we need
relef from the major performance issue that were somehow baked into Version
One.  We can
live with Version One if we have the implementation corresponding to
cm-pvt-msg.  I'm not sure whether this is Grouping Two (as above) or Zero
(because we are are talking about existing implementations).  In any case
we have running code and should take advantage of Chuck's good work in
specifying this regardless of any potential controversy.



On Mon, Jul 31, 2017 at 2:34 PM, Chuck Lever <[e-mail]> wrote:

> Hi-
>
> During the nfsv4 WG meeting at IETF 99, I presented some slides on possible
> next steps for NFS/RDMA and related protocols.
>
> https://datatracker.ietf.org/meeting/99/materials/slides-
> 99-nfsv4-nfsrdma-next-steps-chuck-lever
>
> There are many areas that could use some attention, yet only a handful of
> engineering resources are available. Slides 11 - 18 describe the directions
> that could IMO be fruitful individual next steps.
>
> I hoped my talk could frame a conversation about where we think the highest
> priority is, but it was decided that there were enough interested and
> informed people who were not present that such a conversation should be
> moved to this mailing list and continued after IETF 99.
>
> Note that RFC 5667bis is now "Submitted to IESG for Publication". The work
> items in the slides assume that this document will be completed and
> published
> as planned.
>
> The slides split the possibilities into three somewhat orthogonal
> groupings.
> A fourth grouping arose during discussion in the room, which I'll add as
> "Grouping Zero" below. We can choose any or all of these approaches.
> Opinions
> are welcome as to what order, whether something is left out, or what might
> be
> removed from this list. Groupings Two and Three could introduce new Working
> Group documents, and thus have implications for our charter milestone
> count.
>
>
> Grouping Zero:
> Focus on improving existing implementations of RPC-over-RDMA and NFS/RDMA.
> No
> IETF action needed, which is why I didn't include this on the slides. There
> are substantial improvements that can be made to existing base
> implementations,
> but these would be done by many of the same folks who would be working on
> new
> protocol.
>
>
> Grouping One:
> Enable greater transport parallelism in NFS. This includes multipathing and
> use of pNFS with RDMA. No changes to RPC-over-RDMA or NFS/RDMA are
> necessary,
> and this would bring important performance capabilities to NFS, especially
> by enabling very low latency client access to Storage Class Memory.
>
>
> Grouping Two:
> Incrementally improve RPC-over-RDMA version 1. The main idea here is to
> introduce a per-connection transport property negotiation mechanism to
> replace
> CCP. This would enable variable size (ie larger) inline thresholds and the
> use
> of Remote Invalidation in some instances with existing deployments.
>
>
> Grouping Three:
> Pursue RPC-over-RDMA version 2. This would open a variety of avenues by
> which
> many of the perceived shortcomings of RPC-over-RDMA version 1 could be
> addressed.
>
>
> IMO Zero and One are where we can get the greatest bang for the buck in the
> near term.
>
> Latency to access Storage Class Memory is substantially shorter than the
> latency of traversing the NFS and RPC stack on just the client. Thus
> bypassing RPC entirely (eg by using an RDMA layout type) seems like the
> best
> strategy we have for tapping the potential of this new variety of durable
> storage.
>
> The current proposal for Grouping Two (draft-cel-nfsv4-rpcrdma-cm-pvt-msg)
> is
> controversial. Grouping Three would be an immense amount of work to
> generalize
> some things for less gain than we might see with work in Grouping Zero or
> One.
>
>
> --
> Chuck Lever
>
>
>
> _______________________________________________
> nfsv4 mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/nfsv4
>