Re: [nfsv4] Fwd: New Version Notification for draft-dnoveck-nfsv4-rpcrdma-rtissues-01.txt

[David Noveck <[e-mail]>]

> where with the RDMA_WRITE we do not need to wait? (i.e.
> sections 2.2 and 2.3)

Actually, that is why there is not an internode round trip which
contributes to latency.

I do distinguish in  this documentbetween round-trips which do and do not
contribute to latency.  For example, I mention certain acks which re part
of round trips that nobody has to wait for.

I believe that there is no ack for an RDMA write but I've never actually
looked on the wire to see that.  I imagine that there might be certain rare
cases in which some sort of separate ack were sent.  For example, if you
did multiple RDMA write in sequence without a SEND or there were a long
delay without the response being sent, a separate ACK of the RDMA WRITE
might be sent.

However, in the common case, in which a SEND immediately follows, the RDMA
WRITE, there is no need for a separate ACK and I believe the ACK of the
SEND suffices to assure the responder RNIC, that both
previous operations have completed successfully.

On Thu, Sep 15, 2016 at 2:58 PM, karen deitke <[e-mail]>
wrote:

> Thanks,
>
> That clears things up.  Next question, are you saying that an RDMA_WRITE
> is NOT an internode round trip and an RDMA_READ is an internode round trip,
> because we have to wait for the data from the RDMA_READ before proceeding,
> where with the RDMA_WRITE we do not need to wait? (i.e. sections 2.2 and
> 2.3)
>
> Karen
>
>
> On 9/12/2016 2:24 PM, David Noveck wrote:
>
> > I'm confused by this summarization.
>
> :-(.  Let's see what I can do to make this clearer.
>
> > In the text above you indicate 3 different places where "internode round
> trip is involved, yet in the summary you only mention 2.
>
> The point I was trying to make was that, although there were three
> round-trips, only two contribute to the request latency.  In some
> cases, there is a round trip because an ack is sent, but because neither
> the client nor the server is waiting for it.
>
> > What is the definition of an "internode round trip?"
>
> Any situation in which a message is sent in one direction and, after that,
> another message is sent in the opposite direction.
>
> > Also its unclear to me what you mean my "in the context of a connected
> operation".
>
> maybe I should have said, "Because this reliable connected operation in
> which messages are acked.
>
> > Also you mention that there are two-responder-side interrupt latencies,
> are you referring to the notification of the RDMA_READ
> > and the send completion queue for sending the response?
>
> I'm referring to the notification that the request has been received and
> and the notification that the RDMA_READ has comnpleted.
>
> Does this interrupt latency come into play in the latency of the
> operation?
>
> I think the two I mentioned do.
>
> > Once the client side gets the response it can continue, even if the
> server thread is still waiting for notification of a successful send
> correct?
>
> Yes.
>
> > Also are you missing the interrupt latency of the send on the client? In
> addition to the interrupt latency of receiving the reply?
>
> I don't think that contributes to latency.  The request processing can
> continue once the request is received on the server, even if the client
> has not received notification of the completion of the send.
>
> On Mon, Sep 12, 2016 at 3:52 PM, karen deitke <[e-mail]>
> wrote:
>
>> Hi Dave,
>>
>> I'm struggling following this below:
>>
>>    o  First, the memory to be accessed remotely is registered.  This is
>>       a local operation.
>>
>>    o  Once the registration has been done, the initial send of the
>>       request can proceed.  Since this is in the context of connected
>>       operation, there is an internode round trip involved.  However,
>>       the next step can proceed after the initial transmission is
>>       received by the responder.  As a result, only the responder-bound
>>       side of the transmission contributes to overall operation latency.
>>
>>    o  The responder, after being notified of the receipt of the request,
>>       uses RDMA READ to fetch the bulk data.  This involves an internode
>>       round-trip latency.  After the fetch of the data, the responder
>>       needs to be notified of the completion of the explicit RDMA
>>       operation
>>
>>    o  The responder (after performing the requested operation) sends the
>>       response.  Again, as this is in the context of connected
>>       operation, there is an internode round trip involved.  However,
>>       the next step can proceed after the initial transmission is
>>       received by the requester.
>>
>>    o  The memory registered before the request was issued needs to be
>>       deregistered, before the request is considered complete and the
>>       sending process restarted.  When remote invalidation is not
>>       available, the requester, after being notified of the receipt of
>>       the response, performs a local operation to deregister the memory
>>       in question.  Alternatively, the responder will use Send With
>>       Invalidate and the responder's RNIC will effect the deregistration
>>       before notifying the requester of the response which has been
>>       received.
>>
>>    To summarize, if we exclude the actual server execution of the
>>    request, the latency consists of two internode round-trip latencies
>>    plus two-responder-side interrupt latencies plus one requester-side
>>    interrupt latency plus any necessary registration/de-registration
>>    overhead.  This is in contrast to a request not using explicit RDMA
>>    operations in which there is a single inter-node round-trip latency
>>    and one interrupt latency on the requester and the responder.
>>
>> I'm confused by this summarization.  In the text above you indicate 3
>> different places where "internode round trip is involved, yet in the
>> summary you only mention 2.  What is the definition of an "internode round
>> trip?"  Also its unclear to me what you mean my "in the context of a
>> connected operation".
>>
>> Also you mention that there are two-responder-side interupt latencies,
>> are you referring to the notification of the RDMA_READ and the send
>> completion queue for sending the response?  Does this interrupt latency
>> come into play in the latency of the operation? Once the client side gets
>> the response it can continue, even if the server thread is still waiting
>> for notification of a successful send correct?
>>
>> Also are you missing the interrupt latency of the send on the client? In
>> addition to the interrupt latency of receiving the reply?
>>
>> Karen
>>
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/nfsv4
>>
>
>
>