Re: [nfsv4] [[e-mail]: New Version Notification for draft-hellwig-nfsv4-rdma-layout-00.txt]

[Steve Byan's Lists <[e-mail]>]

> On Jul 19, 2017, at 4:10 AM, Christoph Hellwig <[e-mail]> wrote:
> 
> On Tue, Jul 18, 2017 at 05:33:31PM -0400, Steve Byan's Lists wrote:
> 
>> 2.3. Device Addressing and Discovery
>> 
>> I think addressing and discovery should support multipathing, to enhance
>> availability. So rather than a single netaddr4, I think the
>> struct pnfs_rdma_device_addr4 should be defined to contain a
>> multipath_list4, as in the File layout. 
> 
> Memory registrations are bound to a protection domain, which at least
> for NFS is generally bound to a specific QP, so simply returning
> multiple addresses for interchangable use might not be a good idea.
> 
> It also is a very bad idea for load balancing purposes - I'd much rather
> have the MDS control explicitly which layouts go to which QP, to e.g.
> steer them to different HCAs.  (and not that that nothing in the draft
> requires multiple HCAs used for the RDMA operations to even be in the
> same system).

I think the server may not be the right place for load balancing for 
pNFS-RDMA. The pNFS-RDMA client is much more likely to know if it 
is experiencing congestion delays than even the server HCA/RNIC, much 
less the pNFS-RDMA server software, which is not even involved in the 
data transfer path.

Also, if the client is unable to establish a connection to the single address 
specified by the layout, how can it request the server to fail over to a 
redundant path? Is the server required to hand out a different path 
(assuming it has one) the next time the client requests a layout?

>> It seems better to me to model the extent using strictly file-local
>> information, i.e. the registered handle is simply that resulting from,
>> for example, a user-space server mapping the file, determining its
>> sparseness, and registering the non-sparse extents of the file. Thus
>> re_storage_offset would not be needed in struct pnfs_rdma_extent4. The
>> re_device_id, re_state, re_file_offset, re_length, and a separately
>> provided set of re_handles are sufficient. I view getting a pNFS-RDMA
>> layout as analogous to mmap’ing a local file - the layout mmap’s a
>> file into the RDMA address space of a RC queue pair.
> 
> File mappings are very much an implementation detail.  E.g. one the
> scenarious I want to support with this layout is indirect writes
> where the client gets a write buffer that only gets moved into the
> file itself by the layoutcommit (or RDMA FLUSH/COMMIT operation once
> standardized).
> 
> That would work together with an NFS extension to support O_ATOMIC
> out of place updates ala:
> 
> https://www.usenix.org/conference/fast15/technical-sessions/presentation/verma
> https://lwn.net/Articles/715918/
> 
> to provide byte level write persistent memory semantics over NFS.

I think using an RDMA RPC for atomic writes might be a better approach. 
I’m not convinced that using one-sided RDMA ops is lower latency — the 
client still has to send the layoutcommit RPC. Or if one tries to include
the atomic commit semantic in the proposed RDMA FLUSH/COMMIT,
that effectively turns it into an RPC.

Why not just send the data along with the commit, given that you need 
an RPC anyway?

Also, forcing the interface to the failure-atomic transaction to look like
copy-on-write is awkward for servers that implement it using undo or
redo logging.

It would be good to see some data on the performance of these 
approaches before we bake something into the protocol.

Best regards,
-Steve

-- 
Steve Byan <[e-mail]>
Littleton, MA