Re: [nfsv4] Fwd: New Version Notification for draft-dnoveck-nfsv4-rpcrdma-rtissues-01.txt

[karen deitke <[e-mail]>]

Thanks,


That clears things up.  Next question, are you saying that an RDMA_WRITE 
is NOT an internode round trip and an RDMA_READ is an internode round 
trip, because we have to wait for the data from the RDMA_READ before 
proceeding, where with the RDMA_WRITE we do not need to wait? (i.e. 
sections 2.2 and 2.3)

Karen

On 9/12/2016 2:24 PM, David Noveck wrote:
> > I'm confused by this summarization.
>
> :-(.  Let's see what I can do to make this clearer.
>
> > In the text above you indicate 3 different places where "internode round trip is involved, 
> yet in the summary you only mention 2.
>
> The point I was trying to make was that, although there were three 
> round-trips, only two contribute to the request latency.  In some
> cases, there is a round trip because an ack is sent, but because 
> neither the client nor the server is waiting for it.
>
> > What is the definition of an "internode round trip?"
>
> Any situation in which a message is sent in one direction and, after 
> that, another message is sent in the opposite direction.
>
> > Also its unclear to me what you mean my "in the context of a connected operation".
>
> maybe I should have said, "Because this reliable connected operation 
> in which messages are acked.
>
> > Also you mention that there are two-responder-side interrupt latencies, are you referring 
> to the notification of the RDMA_READ
> > and the send completion queue for sending the response?
>
> I'm referring to the notification that the request has been received 
> and and the notification that the RDMA_READ has comnpleted.
>
> Does this interrupt latency come into play in the latency of the 
> operation?
>
> I think the two I mentioned do.
>
> > Once the client side gets the response it can continue, even if the server thread is still 
> waiting for notification of a successful send correct?
>
> Yes.
>
> > Also are you missing the interrupt latency of the send on the client? In addition to the 
> interrupt latency of receiving the reply?
>
> I don't think that contributes to latency.  The request processing can 
> continue once the request is received on the server, even if the client
> has not received notification of the completion of the send.
>
> On Mon, Sep 12, 2016 at 3:52 PM, karen deitke <[e-mail] 
> <mailto:[e-mail]>> wrote:
>
>     Hi Dave,
>
>     I'm struggling following this below:
>
>        o  First, the memory to be accessed remotely is registered. 
>     This is
>           a local operation.
>
>        o  Once the registration has been done, the initial send of the
>           request can proceed.  Since this is in the context of connected
>           operation, there is an internode round trip involved. However,
>           the next step can proceed after the initial transmission is
>           received by the responder.  As a result, only the
>     responder-bound
>           side of the transmission contributes to overall operation
>     latency.
>
>        o  The responder, after being notified of the receipt of the
>     request,
>           uses RDMA READ to fetch the bulk data.  This involves an
>     internode
>           round-trip latency.  After the fetch of the data, the responder
>           needs to be notified of the completion of the explicit RDMA
>           operation
>
>        o  The responder (after performing the requested operation)
>     sends the
>           response.  Again, as this is in the context of connected
>           operation, there is an internode round trip involved. However,
>           the next step can proceed after the initial transmission is
>           received by the requester.
>
>        o  The memory registered before the request was issued needs to be
>           deregistered, before the request is considered complete and the
>           sending process restarted.  When remote invalidation is not
>           available, the requester, after being notified of the receipt of
>           the response, performs a local operation to deregister the
>     memory
>           in question.  Alternatively, the responder will use Send With
>           Invalidate and the responder's RNIC will effect the
>     deregistration
>           before notifying the requester of the response which has been
>           received.
>
>        To summarize, if we exclude the actual server execution of the
>        request, the latency consists of two internode round-trip latencies
>        plus two-responder-side interrupt latencies plus one requester-side
>        interrupt latency plus any necessary registration/de-registration
>        overhead.  This is in contrast to a request not using explicit RDMA
>        operations in which there is a single inter-node round-trip latency
>        and one interrupt latency on the requester and the responder.
>
>     I'm confused by this summarization.  In the text above you
>     indicate 3 different places where "internode round trip is
>     involved, yet in the summary you only mention 2.  What is the
>     definition of an "internode round trip?"  Also its unclear to me
>     what you mean my "in the context of a connected operation".
>
>     Also you mention that there are two-responder-side interupt
>     latencies, are you referring to the notification of the RDMA_READ
>     and the send completion queue for sending the response?  Does this
>     interrupt latency come into play in the latency of the operation?
>     Once the client side gets the response it can continue, even if
>     the server thread is still waiting for notification of a
>     successful send correct?
>
>     Also are you missing the interrupt latency of the send on the
>     client? In addition to the interrupt latency of receiving the reply?
>
>     Karen
>
>
>     _______________________________________________
>     nfsv4 mailing list
>     [e-mail] <mailto:[e-mail]>
>     https://www.ietf.org/mailman/listinfo/nfsv4
>     <https://www.ietf.org/mailman/listinfo/nfsv4>
>
>