Re: [nfsv4] [[e-mail]: New Version Notification for draft-hellwig-nfsv4-rdma-layout-00.txt]

[David Noveck <[e-mail]>]

> - define the exact connection establishment model.  I'd really
>   like to rely on RDMA/CM for that

Connection establishment can use RDMA/CM to establish an RC
connection.  Unfortunately, as Chuck points out, establishing the
connection is only part of the problem.

> The connection model is critical, because the handles returned
> to clients can work only on one connection (QP).

True :-(

> For example,
> you can't assume that NFS/RDMA will be used to access the MDS,
> nor can you assume that the MDS and DS's are accessed through
> the same HCA/RNIC on the same connection.

Also, you can't assume  there is only a single connection between any
client-DS pair.  Someone will have to choose one and it isn't clear whether
the
client or the MDS is the best one to do the choosing.

> To make it work, there will have to be some way of binding a
> layout (containing handles) to a particular connection to a
> particular storage device.

That sound like it requires a (new) bind operation, separate from the
creation of the
layout.  That is doable but it increases the complexity to define both a
protocol
extension and a pNFS layout type that work together.  Also, IIRC, RFC 8178
may require
a new minor version to do this.

An alternative is for the MDS to do the bind operation as part of assigning
the layout, which
it can do as long as it knows the connection.  One possibility is for the
to client to pass
a description of the chosen connection as a layout-hint.  Another is for
the MDS to be able
to find out the set of possible connections, havie it choose one, and let
the client know,
in the layout, which connection is associated with the layout.

> Also, if the connection to a DS is lost, more than a
> reconnect is necessary. The client will need to take steps to
> get the server to re-register the memory and send fresh
> handles.

The simplest way to deal with this, although maybe not the best, is to
consider the
connection break as  effectively revoking all associated layouts.  The
client needs
to get new layouts to replace the ones lost.

On Mon, Jul 17, 2017 at 5:49 AM, Chuck Lever <[e-mail]> wrote:

>
> > On Jul 3, 2017, at 01:10, Christoph Hellwig <[e-mail]> wrote:
> >
> > As promised here is an initial cut at the RDMA layout before the
> > meeting in Prague.
> >
> > I have to admit it's not the highest quality draft, but I wanted to
> > get it out - it's probably unsuitable for readers without deep
> > knowledge of RDMA at this point.
> >
> > The idea of the layout is to provide RDMA READ / WRITE access to
> > remote memory regions - usually persistent memory in some form,
> > but to some extent it will also work with volatile caching of
> > data, e.g. in features like the NVMe controller memory buffer or
> > even host memory.  It is done by registering these regions on
> > the server and performing the RDMA READ / WRITE operations from
> > the client, that is it inverts the model used by RDMA RPC or
> > other storage models.
> >
> > Besides improving the spec language there still is a lot left to
> > be done:
> >
> > - define the exact connection establishment model.  I'd really
> >   like to rely on RDMA/CM for that
>
> The connection model is critical, because the handles returned
> to clients can work only on one connection (QP). For example,
> you can't assume that NFS/RDMA will be used to access the MDS,
> nor can you assume that the MDS and DS's are accessed through
> the same HCA/RNIC on the same connection.
>
> To make it work, there will have to be some way of binding a
> layout (containing handles) to a particular connection to a
> particular storage device.
>
> Also, if the connection to a DS is lost, more than a
> reconnect is necessary. The client will need to take steps to
> get the server to re-register the memory and send fresh
> handles.
>
>
> > - figure out if we can get rid of the sub-layout extent inherited
> >   from the block layout.  This should be possible by providing two
> >   handles in the layout
> > - find a way to future-proof for the introduction of a RDMA FLUSH
> >   or COMMIT operation, where we don't have to do a LAYOUTCOMMIT
> >   for every write
> >
> > ----- Forwarded message from [e-mail] -----
> >
> > Date: Sun, 02 Jul 2017 16:04:45 -0700
> > From: [e-mail]
> > Subject: New Version Notification for draft-hellwig-nfsv4-rdma-
> layout-00.txt
> > To: Christoph Hellwig <[e-mail]>
> >
> >
> > A new version of I-D, draft-hellwig-nfsv4-rdma-layout-00.txt
> > has been successfully submitted by Christoph Hellwig and posted to the
> > IETF repository.
> >
> > Name:         draft-hellwig-nfsv4-rdma-layout
> > Revision:     00
> > Title:                Parallel NFS (pNFS) RDMA Layout
> > Document date:        2017-07-02
> > Group:                Individual Submission
> > Pages:                18
> > URL:            https://www.ietf.org/internet-
> drafts/draft-hellwig-nfsv4-rdma-layout-00.txt
> > Status:         https://datatracker.ietf.org/
> doc/draft-hellwig-nfsv4-rdma-layout/
> > Htmlized:       https://tools.ietf.org/html/draft-hellwig-nfsv4-rdma-
> layout-00
> > Htmlized:       https://datatracker.ietf.org/
> doc/html/draft-hellwig-nfsv4-rdma-layout-00
> >
> >
> > Abstract:
> >   The Parallel Network File System (pNFS) allows a separation between
> >   the metadata (onto a metadata server) and data (onto a storage
> >   device) for a file.  The RDMA Layout Type is defined in this document
> >   as an extension to pNFS to allow the use of RDMA Verbs operations to
> >   access remote storage, with a special focus on accessing byte
> >   addressable persistent memory.
> >
> >
> >
> >
> > Please note that it may take a couple of minutes from the time of
> submission
> > until the htmlized version and diff are available at tools.ietf.org.
> >
> > The IETF Secretariat
> >
> > ----- End forwarded message -----
> >
> > _______________________________________________
> > nfsv4 mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/nfsv4
>
> --
> Chuck Lever
> [e-mail]
>
>
>
> _______________________________________________
> nfsv4 mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/nfsv4
>