Re: [nfsv4] pNFS file and use of RPC-over-RDMA to access an NFSv4.1 data server

[David Noveck <[e-mail]>]

Thanks to Tom and Chuck for their responses.  I'll respond to both  below.

First of all, let me say how my server will behave on this issue at future
testing events.

I intend to specify both rdma/rdma6 netaddr and a port number of 20049 on
the devinfos associated layouts when the data server supports RDMA.
Clients can then key off either of those two indications or they can use
the fact that the MDS connection is RDMA, if they choose.

I also intend to support use of  the FSLI4TF_RDMA flag as an indication
that an RDMA connection can be used but probably won't have that support on
my intial server.

Whatever the working group deciides to do on this issue, we need to
document the choice in some way.  Maybe we will need a short RFC on this
specfic issue rather than trying to update RFC8267 to include this.   This
short RFC would probably be marked as updating RFC5661, since it does
significantly revise chapter 13 of that document.  I have given some
thought to including this stuff in draft-ietf-nfsv4-mv1-msns-update but so
far it doesn't seem to fit very well.

With regard to Chuck's comments about the analogous problems for referrals,
I have some questions:

   - I assume the same problem can arise with regard to migrations,
   although, for that case it is a reasonable guess, in the absence of any
   other indications that the destination is of the same connection type as
   the source.  Are migration and referral essentially the same in this regard?
   - How useful is  the FSLI4TF_RDMA flag in addressing this problem?  Do
   we need anything else for satisfactory support?
   - What is the situation with regard to NFSv4.0?  How  can  RDMA
   connection be established for migration/referrals in the absence of
   fs_locations_info? Can we regard this as a can't-get-there-from-here
   situation?  If not, how do we address this?

I think that whatever is decided on these questions, this needs to be
addressed in draft-ietf-nfsv4-mv1-msns-update.  Given that this document is
presenting a major respecification of this area, it needs to do so in light
of the fact that multiple connection types are on the agenda now.  Similar
logic might apply in the case of draft-ietf-nfsv4-mv0-trunking-update as
well.  I intend to add discussion of these issues to
draft-ietf-nfsv4-migration-issues-15.

I agree with Tom that human internention is going to be required to deal
with switching/routing problems/incompatibilities although I am not joining
in his "guarantee".   Even though no protocol additions seem likely, we
have to be aware of these difficulties as we draft and review documents in
this area.  As Chuck indicates, the server's and client's knowledge that
they have support are not sufficient to conclude that it is appropriate
that it be used.  This probably means that we will need to avoid most uses
of RFC2119 terms as we describe this area.  It also seems to me that it
would be better if we avoided those terms' lower-case homonyms as well.

With regard to Tom's discussion of SMB3 facilities for presentation of
server interface characteristics, this has been discussed for NFSv4 but it
didn't seem to fit into NFSv4.1.   It is possible to provide such a
facility as an optional extension to NFSv4.2 if someone is interested in
working on this.  Unless there are objections, I will add a discussion of
this possibility to draft-ietf-nfsv4-migration-issues-15.

On Wed, Apr 18, 2018 at 12:07 PM, Tom Talpey <[e-mail]> wrote:

> > -----Original Message-----
> > From: nfsv4 <[e-mail]> On Behalf Of Chuck Lever
> > Sent: Wednesday, April 18, 2018 11:05 AM
> > To: David Noveck <[e-mail]>
> > Cc: NFSv4 <[e-mail]>
> > Subject: Re: [nfsv4] pNFS file and use of RPC-over-RDMA to access an
> NFSv4.1
> > data server
> >
> > Hi Dave-
> >
> > Thanks for bringing this up. I agree it should be explored.
> > A few comments below.
> >
> >
> > > On Apr 18, 2018, at 9:51 AM, David Noveck <[e-mail]>
> > wrote:
> > >
> > > As far as I can determine, there has not been significant discussion
> of the
> > question of how a client which receives a pNFS file layout is to
> determine
> > whether the connection to the data server is to be established using TCP
> or
> > RPC-over-RDMA.  As a result, the Linux NFS client, when it receives a
> file
> > layout, unconditionally establishes a TCP connection to the data
> server.   I’ve
>
> <all good stuff, sniped>
>
> > >     • Expecting the client to connect, at first, using a tcp
> connection and
> > then interrogating the fs_locations_info attribute and using the
> FSLI4TF_RDMA
> > flag as an indication that the client is to access this fs using an
> RDMA-capable
> > transport .
> >
> > This seems like a reliable long-term solution, and it has the
> > benefit of providing not one but a list of interfaces and
> > transport types.
>
> I just want to interject that this is what is done by the server side of
> SMB3
> multichannel, which additionally provides an address and link speed. The
> client then sorts by type and speed, chooses some preferences and performs
> a routing lookup to assess possible connectivity for each type. It then
> proceeds
> to attempt one or more additional connections, moving traffic to them
> if/when
> they succeed.
>
> > However, it falls prey to the issue above, where both ends could
> > support RDMA, but the end-to-end network path might not.
>
> Absolutely yes. And worse, if the connection succeeds but for some reason
> cannot sustain heavy traffic. We see this All The Time with RoCE protocols,
> which depend on accurate PFC and sometimes ECN behaviors to be
> performed by each and every switch hop in the path. The symptoms are
> maddening, and the solution even more so. We hope the RoCE vendors
> will step up to address this over time with diagnosibility information, and
> further refinement of the RoCE protocol requirements. I don't think that
> pNFS or NFSv4 should attempt to address this.
>
> Bottom line, simple is best at the pNFS/NFSv4 layer. I guarantee human
> intervention will always be required, somewhere along the way.
>
> Tom.
>