[nfsv4] pNFS file and use of RPC-over-RDMA to access an NFSv4.1 data server

[David Noveck <[e-mail]>]

As far as I can determine, there has not been significant discussion of the
question of how a client which receives a pNFS file layout is to determine
whether the connection to the data server is to be established using TCP or
RPC-over-RDMA.  As a result, the Linux NFS client, when it receives a file
layout, unconditionally establishes a TCP connection to the data server.
I’ve had to disable pNFS support on the server for my testing, so that the
NFS-over-RDMA data paths can get tested but I’d prefer if we could arrive
at an approach which allows interoperable implementations supporting both
pNFS file and RPC-over-RDMA to be developed and tested.

I have discussed this issue with some people and arrived at an initial list
of ways in which this choice might be made.   Additions are possible but
our principal task is arriving at a compatible subset which can serve as a
basis for use in testing events and eventual standardization and
deployment.  One critical criterion governing our choice is the expected
penalty for making the wrong choice.   When this is high, the need is
strong for a correct decision, while, if we could reduce the penalty, a
reasonable basis for a guess could be an acceptable choice.  In this
connection, it is worth noting that currently Linux’s RoCE support for
RPC-over-RDMA winds up waiting 60 seconds (i.e. about 10 million times the
typical round-trip inter-node delay time) before deciding that
RPC-over-RDMA service is not available.  Until and unless that is changed,
we would need a quite accurate/reliable means of determining when such
support is available.

In any case, here is the list of approaches I have come up with so far.
I’d like the working group to augment the list as new approaches arise and
then decide on a way forward that we can proceed on for use in future
testing events and as a basis for standardization.


   - Deciding that, when the metadata server connection is RDMA, data
   connections are, by default, presumed to be RDMA.   This is a basis for a
   guess and would not be acceptable if the penalty for a wrong guess is
   high.   There will certainly be cases in which RDMA support is present on
   the MDS but might not be available on one or more DS’s.
   - Using the port specified in the layout with 20049 indicating that RDMA
   is to be attempted.   I don’t see any practical issue with this but it is
   kind of hacky.
   - Using a value of   “rdma’ or “rdma6” instead of “tcp” or “tcp6”, in
   the netaddr field of the device  info.  I’m not sure how hard or
   time-consuming it would be to have a new netaddr assigned in the case in
   which we use existing universal address formats.   We need to investigate
   this.
   - Expecting the client to connect, at first, using a tcp connection and
   then interrogating the fs_locations_info attribute and using the
   FSLI4TF_RDMA flag as an indication that the client is to access this fs
   using an RDMA-capable transport .
   - Expecting  the client to connect, at first, using a tcp connection and
   then do a CREATE_SESSION specifying CREATE_SESSION4_FLAG_CONN_RDMA, or to
   address transports in which a within-connection step-up is not possible i.e
   . RoCE or Infiniband, defniing a new similar create-session flag to support
   creation of a new connection to obtain RDMA support on the session using a
   BIND_CONN_TO_SESSION on the new connection.