Return-Path: X-Original-To: nfsv4@ietfa.amsl.com Delivered-To: nfsv4@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E8E5212B10B for ; Mon, 12 Sep 2016 13:24:08 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.699 X-Spam-Level: X-Spam-Status: No, score=-2.699 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5qMfwJNo0aPb for ; Mon, 12 Sep 2016 13:24:06 -0700 (PDT) Received: from mail-it0-x22f.google.com (mail-it0-x22f.google.com [IPv6:2607:f8b0:4001:c0b::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9F8E412B0F8 for ; Mon, 12 Sep 2016 13:24:06 -0700 (PDT) Received: by mail-it0-x22f.google.com with SMTP id 186so27578783itf.0 for ; Mon, 12 Sep 2016 13:24:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=1nogCK+E2ao+uOrmNc9yMoXpFOCYS0oRK9Pl6TLtbtY=; b=QmDK5AaRNd+auh6DJzYA2y6UxAwm4IIFe+DNhXAPzLychSnkLklIwOdd3H/PjMGAZr DNINJFZPW/ndHTO+naTUtuoRHtnMrhMqSaca01vPmF+rvcPDqe8lhmRSaIbsyjyrgGFx xtfMe9mx0W1yDu9JQq5hwykHD3J3mT9qvGtykqL1y62nTyGepuEUdPQ+LzVVfJ9hdSbW d+XvRBFsiZlYgXYsgtr1tREREVxwFXDa0yfRR0uh1Mmr06suCO5FNqAsR4c+7B7omhHs kc1dtvjK4kqGvlzvHvXixuwC5/U3Mj1uDmWFjIT6CVNQh0Zk9XtqhWYtTzJyn6x6yaBh jv8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=1nogCK+E2ao+uOrmNc9yMoXpFOCYS0oRK9Pl6TLtbtY=; b=Yaeu8eij2shWtF/4+ggP8/gpODmBvVJ0w2TpXBtFXAhLwk2vKZSBmISlV2gmSuxcrO d+0HMKP6N0qtWRVmUQVa6gAj5VX88zI9ZJVqA0KF32Mrb6qdnHMy6slIyH3brtptYayQ +SKhqCKyEH8X+pfxKBCKflnMP1T96e/uWEddnqEHfF9y+UrVDrAZNrcfK76RWEQHkIYt X1tZC/ZTXasiLuigOT61k5NCGg/gM/duV1mDUA+/vlihY4o6WtrWQwATCJx9h+mWuLYZ qVPIQPScdOG2CCJp5sZ0bPNtsRp3zZ2yWCAXIarrAOUIQCXrZUovxQr244vcawd2UCOT Gg5Q== X-Gm-Message-State: AE9vXwO5RF1GUCgwBlQL8417DngWHaqwwrFZAs+7F2gq18Kq54T+gXRPDWsQNGlOxd4AnqLYuVYVxOB3lDPJzw== X-Received: by 10.157.12.216 with SMTP id o24mr6392579otd.4.1473711845926; Mon, 12 Sep 2016 13:24:05 -0700 (PDT) MIME-Version: 1.0 Received: by 10.182.192.10 with HTTP; Mon, 12 Sep 2016 13:24:05 -0700 (PDT) In-Reply-To: <234e3071-2b0e-e5a1-f5d5-91919e9388b1@oracle.com> References: <147292013637.2343.7092433187165824743.idtracker@ietfa.amsl.com> <234e3071-2b0e-e5a1-f5d5-91919e9388b1@oracle.com> From: David Noveck Date: Mon, 12 Sep 2016 16:24:05 -0400 Message-ID: To: karen deitke Content-Type: multipart/alternative; boundary=001a113dd80e13eed9053c5547ba Archived-At: Cc: "nfsv4@ietf.org" Subject: Re: [nfsv4] Fwd: New Version Notification for draft-dnoveck-nfsv4-rpcrdma-rtissues-01.txt X-BeenThere: nfsv4@ietf.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: NFSv4 Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Sep 2016 20:24:09 -0000 --001a113dd80e13eed9053c5547ba Content-Type: text/plain; charset=UTF-8 > I'm confused by this summarization. :-(. Let's see what I can do to make this clearer. > In the text above you indicate 3 different places where "internode round trip is involved, yet in the summary you only mention 2. The point I was trying to make was that, although there were three round-trips, only two contribute to the request latency. In some cases, there is a round trip because an ack is sent, but because neither the client nor the server is waiting for it. > What is the definition of an "internode round trip?" Any situation in which a message is sent in one direction and, after that, another message is sent in the opposite direction. > Also its unclear to me what you mean my "in the context of a connected operation". maybe I should have said, "Because this reliable connected operation in which messages are acked. > Also you mention that there are two-responder-side interrupt latencies, are you referring to the notification of the RDMA_READ > and the send completion queue for sending the response? I'm referring to the notification that the request has been received and and the notification that the RDMA_READ has comnpleted. Does this interrupt latency come into play in the latency of the operation? I think the two I mentioned do. > Once the client side gets the response it can continue, even if the server thread is still waiting for notification of a successful send correct? Yes. > Also are you missing the interrupt latency of the send on the client? In addition to the interrupt latency of receiving the reply? I don't think that contributes to latency. The request processing can continue once the request is received on the server, even if the client has not received notification of the completion of the send. On Mon, Sep 12, 2016 at 3:52 PM, karen deitke wrote: > Hi Dave, > > I'm struggling following this below: > > o First, the memory to be accessed remotely is registered. This is > a local operation. > > o Once the registration has been done, the initial send of the > request can proceed. Since this is in the context of connected > operation, there is an internode round trip involved. However, > the next step can proceed after the initial transmission is > received by the responder. As a result, only the responder-bound > side of the transmission contributes to overall operation latency. > > o The responder, after being notified of the receipt of the request, > uses RDMA READ to fetch the bulk data. This involves an internode > round-trip latency. After the fetch of the data, the responder > needs to be notified of the completion of the explicit RDMA > operation > > o The responder (after performing the requested operation) sends the > response. Again, as this is in the context of connected > operation, there is an internode round trip involved. However, > the next step can proceed after the initial transmission is > received by the requester. > > o The memory registered before the request was issued needs to be > deregistered, before the request is considered complete and the > sending process restarted. When remote invalidation is not > available, the requester, after being notified of the receipt of > the response, performs a local operation to deregister the memory > in question. Alternatively, the responder will use Send With > Invalidate and the responder's RNIC will effect the deregistration > before notifying the requester of the response which has been > received. > > To summarize, if we exclude the actual server execution of the > request, the latency consists of two internode round-trip latencies > plus two-responder-side interrupt latencies plus one requester-side > interrupt latency plus any necessary registration/de-registration > overhead. This is in contrast to a request not using explicit RDMA > operations in which there is a single inter-node round-trip latency > and one interrupt latency on the requester and the responder. > > I'm confused by this summarization. In the text above you indicate 3 > different places where "internode round trip is involved, yet in the > summary you only mention 2. What is the definition of an "internode round > trip?" Also its unclear to me what you mean my "in the context of a > connected operation". > > Also you mention that there are two-responder-side interupt latencies, are > you referring to the notification of the RDMA_READ and the send completion > queue for sending the response? Does this interrupt latency come into play > in the latency of the operation? Once the client side gets the response it > can continue, even if the server thread is still waiting for notification > of a successful send correct? > > Also are you missing the interrupt latency of the send on the client? In > addition to the interrupt latency of receiving the reply? > > Karen > > > _______________________________________________ > nfsv4 mailing list > nfsv4@ietf.org > https://www.ietf.org/mailman/listinfo/nfsv4 > --001a113dd80e13eed9053c5547ba Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
> I'm confused by = this summarization.=C2=A0=C2=A0

:-(.=C2=A0 Let'= s see what I=C2=A0can do to make this clearer.

> In the text above you indicate 3 different places where "intern= ode round trip is involved, yet in the summary you only mention 2.=C2=A0=C2= =A0

The point I was trying to make was that, = although there were three round-trips, only two contribute to the request l= atency.=C2=A0 In some=C2=A0
cases, there is a round trip because an ack is sent, but because neither= the client nor the server is waiting for it.

> What is the definition of an "internode round trip?"=C2=A0= =C2=A0

<= div>Any situation in which a message is se= nt in one direction and, after that, another message is sent in the opposit= e direction.

<= /div>
> Also its unclear to me what= you mean my "in the context of a connected operation".

maybe I should have said, "Because this reliabl= e connected operation in which messages are acked.

> = Also you mention that there are two-responder-side interrupt latencies, are= you referring to the notification of the RDMA_READ=C2=A0
= > and the send completion queue for sen= ding the response?=C2=A0=C2=A0

I'm referr= ing to the notification that the request has been received and and the noti= fication that the RDMA_READ has comnpleted.

D= oes this interrupt latency come into play in the latency of the operation?= =C2=A0

<= div>I think the two I mentioned do.=

> Once the client side gets the response it can = continue, even if the server thread is still waiting for notification of a = successful send correct?

Yes.

> Also are you missing the interrupt latency of the send on the cl= ient? In addition to the interrupt latency of receiving the reply?

I don't think that contributes to latency.=C2= =A0 The request processing can continue once the request is received on the= server, even if the client
has not received notification of the completion of the send.

On Mon, Se= p 12, 2016 at 3:52 PM, karen deitke <karen.deitke@oracle.com>= wrote:
Hi Dave,

I'm struggling following this below:

=C2=A0 =C2=A0o=C2=A0 First, the memory to be accessed remotely is registere= d.=C2=A0 This is
=C2=A0 =C2=A0 =C2=A0 a local operation.

=C2=A0 =C2=A0o=C2=A0 Once the registration has been done, the initial send = of the
=C2=A0 =C2=A0 =C2=A0 request can proceed.=C2=A0 Since this is in the contex= t of connected
=C2=A0 =C2=A0 =C2=A0 operation, there is an internode round trip involved.= =C2=A0 However,
=C2=A0 =C2=A0 =C2=A0 the next step can proceed after the initial transmissi= on is
=C2=A0 =C2=A0 =C2=A0 received by the responder.=C2=A0 As a result, only the= responder-bound
=C2=A0 =C2=A0 =C2=A0 side of the transmission contributes to overall operat= ion latency.

=C2=A0 =C2=A0o=C2=A0 The responder, after being notified of the receipt of = the request,
=C2=A0 =C2=A0 =C2=A0 uses RDMA READ to fetch the bulk data.=C2=A0 This invo= lves an internode
=C2=A0 =C2=A0 =C2=A0 round-trip latency.=C2=A0 After the fetch of the data,= the responder
=C2=A0 =C2=A0 =C2=A0 needs to be notified of the completion of the explicit= RDMA
=C2=A0 =C2=A0 =C2=A0 operation

=C2=A0 =C2=A0o=C2=A0 The responder (after performing the requested operatio= n) sends the
=C2=A0 =C2=A0 =C2=A0 response.=C2=A0 Again, as this is in the context of co= nnected
=C2=A0 =C2=A0 =C2=A0 operation, there is an internode round trip involved.= =C2=A0 However,
=C2=A0 =C2=A0 =C2=A0 the next step can proceed after the initial transmissi= on is
=C2=A0 =C2=A0 =C2=A0 received by the requester.

=C2=A0 =C2=A0o=C2=A0 The memory registered before the request was issued ne= eds to be
=C2=A0 =C2=A0 =C2=A0 deregistered, before the request is considered complet= e and the
=C2=A0 =C2=A0 =C2=A0 sending process restarted.=C2=A0 When remote invalidat= ion is not
=C2=A0 =C2=A0 =C2=A0 available, the requester, after being notified of the = receipt of
=C2=A0 =C2=A0 =C2=A0 the response, performs a local operation to deregister= the memory
=C2=A0 =C2=A0 =C2=A0 in question.=C2=A0 Alternatively, the responder will u= se Send With
=C2=A0 =C2=A0 =C2=A0 Invalidate and the responder's RNIC will effect th= e deregistration
=C2=A0 =C2=A0 =C2=A0 before notifying the requester of the response which h= as been
=C2=A0 =C2=A0 =C2=A0 received.

=C2=A0 =C2=A0To summarize, if we exclude the actual server execution of the=
=C2=A0 =C2=A0request, the latency consists of two internode round-trip late= ncies
=C2=A0 =C2=A0plus two-responder-side interrupt latencies plus one requester= -side
=C2=A0 =C2=A0interrupt latency plus any necessary registration/de-registrat= ion
=C2=A0 =C2=A0overhead.=C2=A0 This is in contrast to a request not using exp= licit RDMA
=C2=A0 =C2=A0operations in which there is a single inter-node round-trip la= tency
=C2=A0 =C2=A0and one interrupt latency on the requester and the responder.<= br>
I'm confused by this summarization.=C2=A0 In the text above you indicat= e 3 different places where "internode round trip is involved, yet in t= he summary you only mention 2.=C2=A0 What is the definition of an "int= ernode round trip?"=C2=A0 Also its unclear to me what you mean my &quo= t;in the context of a connected operation".

Also you mention that there are two-responder-side interupt latencies, are = you referring to the notification of the RDMA_READ and the send completion = queue for sending the response?=C2=A0 Does this interrupt latency come into= play in the latency of the operation? Once the client side gets the respon= se it can continue, even if the server thread is still waiting for notifica= tion of a successful send correct?

Also are you missing the interrupt latency of the send on the client? In ad= dition to the interrupt latency of receiving the reply?

Karen


_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www.ietf.org/mailman/listinfo/nfsv4

--001a113dd80e13eed9053c5547ba--