Re: [nfsv4] Fwd: New Version Notification for draft-dnoveck-nfsv4-rpcrdma-rtissues-01.txt

[karen deitke <[e-mail]>]

Hi Dave,

I'm struggling following this below:

    o  First, the memory to be accessed remotely is registered.  This is
       a local operation.

    o  Once the registration has been done, the initial send of the
       request can proceed.  Since this is in the context of connected
       operation, there is an internode round trip involved.  However,
       the next step can proceed after the initial transmission is
       received by the responder.  As a result, only the responder-bound
       side of the transmission contributes to overall operation latency.

    o  The responder, after being notified of the receipt of the request,
       uses RDMA READ to fetch the bulk data.  This involves an internode
       round-trip latency.  After the fetch of the data, the responder
       needs to be notified of the completion of the explicit RDMA
       operation

    o  The responder (after performing the requested operation) sends the
       response.  Again, as this is in the context of connected
       operation, there is an internode round trip involved.  However,
       the next step can proceed after the initial transmission is
       received by the requester.

    o  The memory registered before the request was issued needs to be
       deregistered, before the request is considered complete and the
       sending process restarted.  When remote invalidation is not
       available, the requester, after being notified of the receipt of
       the response, performs a local operation to deregister the memory
       in question.  Alternatively, the responder will use Send With
       Invalidate and the responder's RNIC will effect the deregistration
       before notifying the requester of the response which has been
       received.

    To summarize, if we exclude the actual server execution of the
    request, the latency consists of two internode round-trip latencies
    plus two-responder-side interrupt latencies plus one requester-side
    interrupt latency plus any necessary registration/de-registration
    overhead.  This is in contrast to a request not using explicit RDMA
    operations in which there is a single inter-node round-trip latency
    and one interrupt latency on the requester and the responder.

I'm confused by this summarization.  In the text above you indicate 3 
different places where "internode round trip is involved, yet in the 
summary you only mention 2.  What is the definition of an "internode 
round trip?"  Also its unclear to me what you mean my "in the context of 
a connected operation".

Also you mention that there are two-responder-side interupt latencies, 
are you referring to the notification of the RDMA_READ and the send 
completion queue for sending the response?  Does this interrupt latency 
come into play in the latency of the operation? Once the client side 
gets the response it can continue, even if the server thread is still 
waiting for notification of a successful send correct?

Also are you missing the interrupt latency of the send on the client? In 
addition to the interrupt latency of receiving the reply?

Karen