Return-Path: X-Original-To: nfsv4@ietfa.amsl.com Delivered-To: nfsv4@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4B6A8126C0F for ; Mon, 31 Jul 2017 11:47:08 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.699 X-Spam-Level: X-Spam-Status: No, score=-2.699 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6ifVoZFEFtLJ for ; Mon, 31 Jul 2017 11:47:06 -0700 (PDT) Received: from mail-it0-x22a.google.com (mail-it0-x22a.google.com [IPv6:2607:f8b0:4001:c0b::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 134B112940A for ; Mon, 31 Jul 2017 11:47:06 -0700 (PDT) Received: by mail-it0-x22a.google.com with SMTP id m34so2495841iti.1 for ; Mon, 31 Jul 2017 11:47:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=F5afudSmwmtGWBwMwbgKlKncUyNR5SCrT22fEeGAvus=; b=PLBebc/4cKFjCmrhON5ztVPRSmN79xZ4a7KAwPeqBSq4KegEakFIGsuYJFBO1hx24X TVEXWtJL9Z4jOoIGgks+ACjhKj3DN/DKv9OxHFMeswogocoXnHYP9y+Pub6a9IJj++PJ hP5N2c1QibUmuiYNNx5QiqPuS+ACdKyYTwXxzpExB51VkNLiUQ3gwghtWIH5kJNQmqjz jYbJLOI5UrwgVKAXsHfZxACth/bmF+X5Po5Hz0Jc5VKbpkVfem6/F/j9+heSuOZNQGKF U7qvuUXo0JeOh8Bthh1TXm6BbW25v53oqqeQWPyjZ2Yi3Ws7AelnV0ESZCG9reypfmVN oICg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=F5afudSmwmtGWBwMwbgKlKncUyNR5SCrT22fEeGAvus=; b=bNgKd65rauCkMgtoekdubw2QyCYUxsSfrriuVki+SuN5Ygo1zLwQ5EJv7Dm2zO5GS4 UEAJr2vkefA8quSt4UC5ag19s0E8y2FM+V3GFnK8BQwA2RtB8pG7UL8pLZeBHBPHyYeg qLYLkDrjsYAZAdvu56d5b/DjGu9f9ufkz1qVHrzYLJznD+A4TNyfE6yUqOLIhZP06C5V bCHaGLVK8jCmxYND5eeWObtKWfR4smI20CNxIE4MZrrVi6iZrJK9WqNuPuC7M4clh7KI rWXjuW07wiEEvPUVHXxb4Mg/nzs+3Zmo7etd3pwS/v1FUtu4w6uT2E5m+/mNml76MtT4 dDbg== X-Gm-Message-State: AIVw1135eLwHRmOU3ITRaRzX/n7ns41L/kFddz6EKmmjqcXe4S9dcRk4 /uXkoVcI+qx4EcWq8ODHHD7/qvujbA== X-Received: by 10.36.253.71 with SMTP id m68mr18497038ith.16.1501526824952; Mon, 31 Jul 2017 11:47:04 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.142.72 with HTTP; Mon, 31 Jul 2017 11:47:04 -0700 (PDT) From: David Noveck Date: Mon, 31 Jul 2017 14:47:04 -0400 Message-ID: To: Chuck Lever , "nfsv4@ietf.org" Content-Type: multipart/alternative; boundary="94eb2c11ce4a05aa980555a17522" Archived-At: Subject: [nfsv4] Credit management and one-way messages X-BeenThere: nfsv4@ietf.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: NFSv4 Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Jul 2017 18:47:08 -0000 --94eb2c11ce4a05aa980555a17522 Content-Type: text/plain; charset="UTF-8" One issue that has been mentioned as a possible impediment to advnacing RPC-over-RDMA to be a working group document, in its current form, concerns its use of one-way messages to convey transport properties. These concerns seem to arise from the presentation of credit managemt as a "request-grant protocol". If the credit logic in fact depended on that pairing, then there would be no roon for one-way meesages. In fact, it doesn't. If you look at the actual operation of credit management, one sees a different picture from what is stated in the introductory paragraph of Section 3.1.1 of RFC 8166: Flow control for RDMA Send operations directed to the Responder is implemented as a simple request/grant protocol in the RPC-over-RDMA header associated with each RPC message. what follows focuses on the role of the grant. giving extensive information but how it is computed and nd why it is vsli for the requester to use it. While the text says "Practically speaking, the critical value is the granted value", it is not clear what the exact role of the request is. It aapears to be a hintwhich the receiver is under no obligation to take any notice of. So it appears that the presentation of the creddit management approach as a request-grant protocol., is a reflection of the fact that, when it is used to transport RPC Requests and Replies, this pairing always exists. When grants are sent with one-way messages, there is no problem that arises from the fact that there is no corresponding credit request. The sender simply informs the receiver about his own receive resources and thus his ability to accept further sends. There is one potential issue connected with use of one-way messages to be addressed. If the recever has N credits and then N one-way messages are sent without any traffic in the opposite direction, then it is possble for a deadlock to result, since there would be no way for the sender to find out about the receive resources. For most one-way messages, this is not a problem, since many one-way messages naturally give rise to messages in the opposite direction even if the relationship is not formalized witin an RPC paradigm. For example: - The RDMA2_CONNPROP sent by the client to the server is paired with an RDMA2_CONNPROP in the opposite direction. - An RDMA2_REQPROP results in RDMA2_RESPROP send in response. RDMA2_UPDPROP is an exception. In the unlikely event that there are a a lage series of such messages sent in one irection while there are no RPC's being sent to the receiver it is possible for a ealock to arise. This would result in a situation in which the receiver would have to send some message back in the other to provide a credit grant. The most likely way to to that is for it to choose to send an RDMA2_UPDPROP with an empty property set in the reverse direction. One other problem with one-way messages as they stand in rpcrdma-version-two concerns section 7.2.2 in which the following item in the bullet list is poblematic: - When the rdma_proc field has the value RDMA2_OPTIONAL and no RPC message payload is present, a Requester MUST set the value of the rdma_optdir field to CALL, and a Responder MUST set the value of the rdma_optdir field to REPLY. The Requester chooses a value for the rdma_xid field from the XID space that matches the message's direction. Requesters and Responders set the rdma_credit field in a similar fashion: a value is set that is appropriate for the direction of the message. This cannot be acted on, because, in the context of a one-way message, it is not clear which party is the requester and which is the responder. While the roles of client and server are fixed and clear, the roles of requester and responder vary from RPC to RPC. If you are not in an RPC context, then any decision as to who is the requestor or responder is arbitrary. I think the best way to address these issues is for rpcrdma-version-two would be to: - Provide an explanation of credit management not so tied to the RPC paradigm. - Add a new value for the direction field for one-way messages - Provide that one-way messages always contain a credit grant rather than a creit request - Explan how the potential deadlock with RDMA2_UPDPROP can be avoided. --94eb2c11ce4a05aa980555a17522 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
One issue that has been mentioned as a possible impediment= to advnacing RPC-over-RDMA to be a working group document, in its current = form, concerns its use of one-way messages to convey transport properties.= =C2=A0 These concerns seem to arise from the presentation of credit managem= t as a "request-grant protocol".=C2=A0 If the credit logic in fac= t depended on that pairing, then there would be no roon for one-way meesage= s.

In fact, it doesn't. If you look at the actual op= eration of credit management, one sees a different picture from what is sta= ted in the introductory paragraph of Section 3.1.1 of RFC 8166:
<= span style=3D"color:rgb(0,0,0)">
Flow control for RDMA Send operations directed to the Responder is= =C2=A0implemented as a simple request/grant protocol in the RPC-over-RD= MA=C2=A0header associated with each RPC message.

what follows focuses on the role of the grant. giving extensive in= formation but how it is computed and nd why it is vsli for the requester to= use it.

While the text says "Practically speaking, the critical value is the granted value",= it is not clear what the exact role of the request is.=C2=A0 It aapears to= be a hintwhich the receiver is under no obligation to take any notice of.= =C2=A0 So it appears that the presentation of the creddit management approa= ch as a request-grant protocol., is a reflection of the fact that, when it = is used to transport RPC Requests and Replies, this pairing always exists.= =C2=A0 When grants are sent with one-way messages, there is no problem that= arises from the fact that there is no corresponding credit request.=C2=A0 = The sender simply informs the receiver about his own receive resources and = thus his ability to accept further sends.

There is one potential issue connect= ed with use of one-way messages to be addressed.=C2=A0 If the recever has N= credits and then N one-way messages are sent without any traffic in the op= posite direction, then it is possble for a deadlock to result, since there = would be no way for the sender to find out about the receive resources.=C2= =A0 For most one-way messages, this is not a problem, since many one-way me= ssages naturally give rise to messages in the opposite direction even if th= e relationship is not formalized witin an RPC paradigm.=C2=A0 For example:<= /div>
  • The RDMA2_CONNPROP sent by the client to the server is pa= ired with an RDMA2_CONNPROP in the opposite direction.
  • An RDMA2_REQPROP results in RDMA2_RESPROP=
 send in response.
RDMA= 2_UPDPROP is an exception. In the unlikely event that there are a a lage s= eries of such messages sent in one irection while there are no RPC's be= ing sent to the receiver it is possible for a ealock to arise. This would = result in a situation in which the receiver would have to send some message= back in the other to provide a credit grant. The most likely way to to t= hat is for it to choose to send an RDMA2_UP= DPROP with an empty property set in the reverse direction.

One other = problem with one-way messages as they stand in rpc= rdma-version-two concerns section 7.2.2 in which the following item in the = bullet list is poblematic:
  • When the rdma_proc field has the value RDMA2_OPTIONAL a=
nd no RPC message payload is present, a Requester MUST set the value of the=
 rdma_optdir field to CALL, and a Responder MUST set the value of the rdma_=
optdir field to REPLY.  The Requester chooses a value for the rdma_xid fiel=
d from the XID space that matches the message's direction.  Requesters =
and Responders set the rdma_credit field in a similar fashion: a value is s=
et that is appropriate for the direction of the message.
    • This cannot be acted on,=
 because, in the context of a one-way message, it is not clear which party =
is the requester and which is the responder.  While the roles of client and=
 server are fixed and clear, the roles of requester and responder vary from=
 RPC to RPC.  If you are not in an RPC context, then any decision as to who=
 is the requestor or responder is arbitrary.
<= span style=3D"white-space:pre-wrap;color:rgb(0,0,0);font-family:arial,helve= tica,sans-serif">I think the best way to address these issues is for rpcrdm= a-version-two would be to:
  • Provide an explanation of credit management not so tied to the RPC paradig= m.
  • Add a new value for the di= rection field for one-way messages
  • Provide that one-way messages always contain a credit grant rather th= an a creit request
  • Explan how= the potential deadlock with RDMA2_UPDPROP can be avoided.

--94eb2c11ce4a05aa980555a17522--