Re: [nfsv4] New draft for working group charter

[Rick Macklem <[e-mail]>]

David Noveck wrote:
[stuff snipped]
> It is not that you missed the mark.  It is just when i saw
> "legacy storage", I thought "spinning metal".
>
> I'm thinking that a better way to say this might be:
>
> Provide NFSv4 to files stored using other file access
> protocol, through the definiton of appropriate pNFS mapping
> types.
I'm guessing that by "mapping types" you are referring to "layout types"?
If I'm guessing, maybe others would be too.

rick
[lots of stuff too hard to delete in this web mail system, so I'm leaving it here;-)
On Sat, May 13, 2017 at 12:35 PM, Chuck Lever <[e-mail]<mailto:[e-mail]>> wrote:

> On May 13, 2017, at 7:28 AM, David Noveck <[e-mail]<mailto:[e-mail]>> wrote:
>
> > Yes, NFS/RDMA is a performance play, but IMO performance
> > is orthogonal here.
>
> Yes.  The motivation and the mechanism are orthogonal.
>
> > The WG's role in the RPC-over-RDMA Version One work was
> > clarification to produce refreshed standards. The WG is
> > responsible for ensuring correct interoperation. The WG
> > did not do any implementation or measurement, and did not
> > ask for a performance-related justification before I
> > started this work.
>
> True.
>
> > We'll have to agree to disagree on this.
>
> I'm not sure exactly what we disagree about.
>
> I guess we have to agree to disagree about whether we
> disagree or not :-)
>
> > Extension is a means to an end that is mediated by WG
> > actions. There can't be a protocol extension without
> > interaction with the WG.
>
> and the IESG and the RFC editor.
>
> > Performance work is as much about implementation as it
> > is about protocol.
>
> The part that is about implementation is generally not the
> business of the WG.  However, there are parts that involve
> protocol changes and the WG does need to be involved in those.
>
> > Performance needs are discovered
> > outside the WG, and that is also where new protocol is
> > frequently invented.
>
> Generally, needs are are discovered outside the WG, nnd
> brought to the attentIon of the working group by one more
> people authoring documents.  That applied to performnce
> with Tom and Brent's RDMA work in the same way it applied
> to the non-performance needs behind umask.
>
> This applies to extensions, new transports like RPC-over-RDMA,
> or new versions.  The WG is a means to an end since none
> of these things can happen without the WG.
>
> > The WG's role in performance work is IMO much more tenuous.
>
> I don't see the difference.
>
> >> I don't see the point of doing that. I don't expect the IESG to
> >> care very much.  This is the kind of thing I would add if it
> >> seemed that the charter was too short.
>
> > It's not filler, IMO.
>
> You're right.  It isn't.  My point was that I wouldn't really expect its absence
> to be an issue for the IESG.
>
> > It's almost less than a single
> > sentence.
>
> I think it would be exactly one sentence.
>
> > Who would approve change requests on those
> > registries if the WG were dissolved?
>
> It would probably be TSWG, in that unlikely event.
>
> OTOH, if this were not in the charter, I wouldn't expect us to be prevented from doing this.  It wasn't in the previous charter.
>
> > I don't see any harm in adding this, considering there
> > has been activity in this area since the charter was
> > last updated.
>
> I agree there will no harm.  I will put the sentence you want to see in the
> Maintenance section of the next draft.
>
> > I think I'd like you to consider the expanded summary
> > of WG member activity I proposed above, instead of a
> > section that focuses solely on performance work.
>
> I can move in that direction (see below) but I think the
> charter should not summarize all areas that we might work on.
> There needs to be room for extensions not mentioned in the
> charter.  I agree to not having a specific performance section,
> but we might expand the Extension section to include a list
> of the major areas that will drive extension work.
>
> From you list:
>
> > enabling the performance attributes of advanced network
> > fabrics and new storage technologies,
>
> OK.
>
> > enabling use of NFS in warehouse virtualization environments,
>
> I don't object to this but I'm not clear what work is beIng referred to.
> Please clarify.

NFSv4.2 features such as ALLOCATE and SEEK_HOLE and space
reservations and S2SC. These are very useful when preparing
or moving virtual disk images, or when thin provisioning is
in play.

The re-treading of NFS/RDMA. NFS/TCP is typically slow in
guests, but NFS/RDMA works in guests as well as NFS/TCP
works on bare metal.


> > helping NFS adapt
> > to increasing network security challenges,
>
> OK.
>
> > and extending
> > the life of legacy storage by enabling pNFS access to it?
>
> I"m not sure what this refering to either.

The pNFS flexfiles layout type. Feel free to adjust this
item if you feel my simple summary misses the mark.


> On Fri, May 12, 2017 at 12:38 PM, Chuck Lever <[e-mail]<mailto:[e-mail]>> wrote:
>
> > On May 12, 2017, at 9:54 AM, David Noveck <[e-mail]<mailto:[e-mail]>> wrote:
> >
> >
> > > I'm wondering why there is now a special charter focus on
> > > performance, but not on virtualization, which has been a
> > > de facto focus of WG effort for the past five years.
> >
> > Let's be clear on the context and stress that what I posted is my draft.  I posted
> > it to facilitate discussion.  It is not posted on a this-is-the-answer basis or a
> > take-it-or-leave-it basis.  It is posted on a suggest-a-change-or-an-
> > alternative basis.
> >
> > > Why isn't it enough to say "The WG is responsible for
> > > extending these protocols as needed."
> >
> > I think we could get by with that but let me explain why I
> > had a specific section in my draft.
> >
> > I think that a draft without this section would not give the IESG
> > a fair picture of the work we would be doing.
>
> In that case, how about instead of a section dedicated to
> just performance, there is a section that summarizes all
> areas where WG members are focusing, which includes
> enabling the performance attributes of advanced network
> fabrics and new storage technologies, enabling use of NFS
> in warehouse virtualization environments, helping NFS adapt
> to increasing network security challenges, and extending
> the life of legacy storage by enabling pNFS access to it?
>
>
> > Let's consider
> > the likely agenda for our meeting at IETF99.
> >
> > If you exclude the charter discussion, over half the minutes will
> > be taken up by performance-related sessions: 20 minutes for
> > you to "drone" :-), 40 minutes for Christoph's pNFS mapping types,
> > and probably 15 minutes devoted to trunking-related stuff.   This
> > is over 60% devoted to performance-related work.  Right now, it
> > is what we are doing, for good reasons.
> >
> > A lot of these things will not have milestones in the draft we submit.
> > In that case we will have to add milestones afterwards.  it is likely
> > that Spencer D would approve those (as extensions) even if there
> > were not a specific performance section.  However, I am worried
> > that if there is no performance section in the charter, the IESG
> > might wonder about us straying from the agreed charter in devoting
> > so much effort to performance.
> >
> > We might get pushback on the performance section from the IESG,
> > and if we get that we can drop it.  However I'd prefer us to explain now
> > the reasons we expect a considerable focus on performance over the
> > next few years.
> >       • For a long while, due to some fortunate circumstances, we have been behind the performance curve and have struggled to catch up.
> >       • Just as our efforts to do so have started to bear fruit, there is another likely burst of technological innovation, in the form of storage class memory, that will up the performance ante, requiring the WG to spend further efforts on the performance front.
> > > If there is a constraint on what may be included in our
> > > charter, it seems like there will be a problem including
> > > performance in particular, as there are no WG documents
> > > specifically about performance.
> >
> > In fact, you just spent a while working on three of them.  Clearly,
> > the basic reason for RPC-over-RDMA is performance, even if the
> > title is not anything like the Trumpian "Really Huge NFS
> > Performance" :-)
>
> Yes, NFS/RDMA is a performance play, but IMO performance
> is orthogonal here.
>
> The WG's role in the RPC-over-RDMA Version One work was
> clarification to produce refreshed standards. The WG is
> responsible for ensuring correct interoperation. The WG
> did not do any implementation or measurement, and did not
> ask for a performance-related justification before I
> started this work.
>
> We'll have to agree to disagree on this.
>
>
> > > If protocol innovation is indeed driven by the individuals
> > > on the WG,
> >
> > It is.
> >
> > > then it seems to me that "performance" should
> > > not be part of our charter.
> >
> > I don't see the logic here.  Extensions are also driven by
> > individuals in/on the WG, yet that is in the charter.  Putting
> > something in the charter does not imply that this will be
> > driven by someone other than the individuals in the WG.
>
> Extension is a means to an end that is mediated by WG
> actions. There can't be a protocol extension without
> interaction with the WG.
>
> Performance work is as much about implementation as it
> is about protocol. Performance needs are discovered
> outside the WG, and that is also where new protocol is
> frequently invented. The WG's role in performance work
> is IMO much more tenuous.
>
>
> > > Improving performance should
> > > be facilitated by the WG / IETF partnership, but driven by
> > > the diaspora.
> >
> > It should be driven by people who have ideas and are willing to
> > explain and implement them, wherever they happen to live.  I
> > hope the IETF will facilitate that.
> >
> > > I'm comfortable with the Maintenance and Extension sections,
> >
> > Good.  It appears that Spencer S. is not comfortable
> > with the latter although he has not proposed an alternative.
> >
> > > and might even add -- for completeness -- that the WG is
> > > responsible for approving changes to certain sections of
> > > the IANA registries.
> >
> > I don't see the point of doing that. I don't expect the IESG to
> > care very much.  This is the kind of thing I would add if it
> > seemed that the charter was too short.
>
> It's not filler, IMO. It's almost less than a single
> sentence. Who would approve change requests on those
> registries if the WG were dissolved?
>
> I don't see any harm in adding this, considering there
> has been activity in this area since the charter was
> last updated.
>
>
> > > I am in philosophical agreement with the technical goal of
> > > better performance, and will continue working on it.
> >
> > Good.  Your work so far has put us in a much better position
> > and I hope we can maintain the momentum.
> >
> >
> > > Still not sure whether that is something that needs to appear in
> > > the WG charter.
> >
> > Is it fair to characterize your current position on this as "No Objection"?
>
> I think I'd like you to consider the expanded summary
> of WG member activity I proposed above, instead of a
> section that focuses solely on performance work.
>
>
> > If so, I'll leave the performance section in for now and we can resolve
> > this question at IETF99.
> >
> >
> >
> > On Fri, May 12, 2017 at 8:56 AM, David Noveck <[e-mail]<mailto:[e-mail]>> wrote:
> > > cm-pvt-msg could move forward, but there has been a palpable
> > > objection to making this work an official standard. Thus I
> > > don't feel it is ready to promote without further discussion.
> >
> > I wasn't suggesting that it move forward without further discussion.
> > I was suggesting that we could well the have necessary discussion
> > by IETF-99.
> >
> > My understanding is that the objection is to it being a standards-track
> > document so that it could move forward as a WG Experiental
> > document and so get a milestone.
> >
> > I think this is right as a Proposed Standard but it isn't really worth
> > spending a lot of time arguing about the matter.  The
> > important facts are that there is an implementation of this
> > in the Linux client, and that it could help address two important
> > performance issues in RPC-over-RDMA Version One: invalidation
> > overhead and excessive use of reply chunks.   As a result, server
> > implementers that care about performance will implement it,
> > whether it is a Proposed Standard, an Experimental RFC, an
> > Informational RFC, a work-in-progress, or an expired individual
> > submission.
> >
> > I think it is best for us to have the discussion and move the
> > document  forward with whatever status the working group
> > can agree on.
> >
> > > RPC-over-RDMA Version Two could become a large project.
> >
> > It doesn't have to be.  As I recall, you proposed, with good
> > reason, a very small Version Two.
> >
> > As far as I'm concerned Version Two, as defined in draft-cel-rpcrdma-version-two
> > is essentially Version One Done Right.   It is designed so that it is easy to
> > create an implementation that interoperates with Version One peers.  The
> > changes are small and limited to:
> >       • A larger default inline threshhold.
> >       • A simple mechanism to exchange information about transport properties
> >       • The ability to define OPTIONAL extensions
> >       • A more flexible remote invalidation scheme
>
> - There are significant improvements to error reporting.
>
> - We need to figure out how to deal with one-way messages
> in a protocol that uses request-grant credit management,
> or drop the use of one-way messages.
>
> - There needs to be new properties that describe transport
> header size limits, chunk list length limits, segment
> size limits, and so on.
>
> - If V2 can take more time to mature, I would like to
> consider ways to eliminate the need for reply size
> estimation. I have a mechanism in mind that also can
> reduce the frequency of Reply chunks.
>
> - There are currently no V2 prototypes.
>
> In other words, the scope of V2 has already stretched,
> and what's more, it feels like more oak-barrel aging is
> needed.
>
>
> > > There are some issues that the WG needs to consider
> > > carefully before enabling such an effort, most especially
> > > what is the appetite by vendors/implementers for a new
> > > version of the protocol, but also, who has the resources
> > > to construct prototypes?
> >
> > I think the only way to find out is to ask people and the best way
> > to do that is to discuss the issue of moving this forward.  If  we had
> > that discussion and the working group was supportive, I would
> > probably look, when I implemented a Version One server, how
> > much work it would take to make it compatible with Version Two
> > as well.   I think that information might be helpful to people.
> >
> > I think we should just let the working group decide this issue.
>
> Yes, I'd like more WG discussion. I'm simply not putting
> a time limit on it because there seems to be a lot more
> work to be done. Like you said above, if the WG reaches
> some consensus about this later, it can be added to the
> charter.
>
> Or we can change the admittance criteria for the charter,
> and let this in now, without a WG document. It seems to
> me if the WG agrees to change the charter to include
> some item, that is enough consensus to assume that there
> will be some work done in that area. Perhaps having an
> extant WG document is too high a bar?
>
>
> > if we
> > can't do that by IETF99, we might be able to do so by the Fall
> > Bakeathon.  I don't see what kind of advance consideration you
> > are anticipating having before proceeding to have the working
> > group discussion.server.
> >
> > > Do we understand and agree on
> > > what problems we are trying to address in the longer term?
> >
> > No we don't but that is the reason for an extensible protocol.
> > We need that sort of agreement when we decide on what
> > extensions to develop.  I agree that that decision is a ways off.
> >
> > > Have we reached the end of RoRV1's ability to deliver
> > > performance?
> >
> > Probably not.  There is a lot of implementation work to be done, but if
> > people do find bottlenecks that require an extension, it is better to be
> > prepared.  The reason for Version Two is to be able to make limited
> > changes when necessary.  It will not make performance improvements
> > on its own, except perhaps in environments that could benefit from the
> > more general remote invalidation support.
>
> My point is that there is a limited pool of resources to
> work on both implementations and protocol development.
> Any work done on V2 means there is something that falls
> off of some part of the V1 plate.
>
> Now that the V1 documents are reaching maturity and
> stability, many of us would like to focus on improving
> our V1 implementations, and move ahead with the new
> pNFS layout type ideas.
>
> That doesn't mean V2 work can't happen, but I don't like
> the idea of treating it like there's a giant vacuum that
> has to be filled right now with new protocol work.
>
>
> > > There have been questions about the scope of RoRV2 for a
> > > while.
> >
> > I recall haring claims that a big Version Two is needed but I
> > haven't heard them for a while.  If there are still people
> > advocating a large/massive Version Two, we can have the
> > necessary discussion as part of proposing advancing our
> > minimal Versiion Two.
> >
> > > The answer shifts depending on whether we think
> > > there is or is not a suitable out-of-band mechanism for
> > > enabling Remote Invalidation and large inline thresholds.
> >
> > I don't think that's the case.  I think the question of
> > the out-of-band mechanism affects the urgency of
> > proceeding with a minimal Version Two.
>
> I think I said that.
>
>
> > Put another way, if cm-pm-msg had never existed, the
> > Version Two would be an urgent matter to be addressed
> > immediately.  Given that it does exist, I think we can take
> > a more leisurely approach and look to make this decision
> > in the next six months.
>
> A more leisurely approach means we have room to consider
> a wider variety of approaches for V2. Unless we explicitly
> decide not to.
>
> The tl;dr is there needs to be an explicit discussion
> revisiting the scope of V2, after the disposition of
> cm-pvt-msg is decided, if only to confirm that the
> scope remains the same.
>
>
> > > To promote this document I would like to have clarification
> > > of the scope of RoRV2; and that in turn requires clarity on
> > > the disposition of cm-pvt-msg.
> >
> > I don't see the need but I think the clarity is availble.  The formalites
> > don't really matter.  What matters is running code and it appears that
> > the running code provides the mechanism you are looking for, and it
> > has been and will be implemented.  I don't see how that affects the
> > scope of Version Two.  What is in rpcrdma-version-two seems a
> > good fit for our current needs in any case.
>
> --
> Chuck Lever
>
>
>
>

--
Chuck Lever