[nfsv4] Credit management and one-way messages

David Noveck <davenoveck@gmail.com> Mon, 31 July 2017 18:47 UTC

Return-Path: <davenoveck@gmail.com>
X-Original-To: nfsv4@ietfa.amsl.com
Delivered-To: nfsv4@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id 4B6A8126C0F for <nfsv4@ietfa.amsl.com>; Mon, 31 Jul 2017 11:47:08 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.699
X-Spam-Status: No, score=-2.699 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id 6ifVoZFEFtLJ for <nfsv4@ietfa.amsl.com>; Mon, 31 Jul 2017 11:47:06 -0700 (PDT)
Received: from mail-it0-x22a.google.com (mail-it0-x22a.google.com [IPv6:2607:f8b0:4001:c0b::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 134B112940A for <nfsv4@ietf.org>; Mon, 31 Jul 2017 11:47:06 -0700 (PDT)
Received: by mail-it0-x22a.google.com with SMTP id m34so2495841iti.1 for <nfsv4@ietf.org>; Mon, 31 Jul 2017 11:47:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=F5afudSmwmtGWBwMwbgKlKncUyNR5SCrT22fEeGAvus=; b=PLBebc/4cKFjCmrhON5ztVPRSmN79xZ4a7KAwPeqBSq4KegEakFIGsuYJFBO1hx24X TVEXWtJL9Z4jOoIGgks+ACjhKj3DN/DKv9OxHFMeswogocoXnHYP9y+Pub6a9IJj++PJ hP5N2c1QibUmuiYNNx5QiqPuS+ACdKyYTwXxzpExB51VkNLiUQ3gwghtWIH5kJNQmqjz jYbJLOI5UrwgVKAXsHfZxACth/bmF+X5Po5Hz0Jc5VKbpkVfem6/F/j9+heSuOZNQGKF U7qvuUXo0JeOh8Bthh1TXm6BbW25v53oqqeQWPyjZ2Yi3Ws7AelnV0ESZCG9reypfmVN oICg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=F5afudSmwmtGWBwMwbgKlKncUyNR5SCrT22fEeGAvus=; b=bNgKd65rauCkMgtoekdubw2QyCYUxsSfrriuVki+SuN5Ygo1zLwQ5EJv7Dm2zO5GS4 UEAJr2vkefA8quSt4UC5ag19s0E8y2FM+V3GFnK8BQwA2RtB8pG7UL8pLZeBHBPHyYeg qLYLkDrjsYAZAdvu56d5b/DjGu9f9ufkz1qVHrzYLJznD+A4TNyfE6yUqOLIhZP06C5V bCHaGLVK8jCmxYND5eeWObtKWfR4smI20CNxIE4MZrrVi6iZrJK9WqNuPuC7M4clh7KI rWXjuW07wiEEvPUVHXxb4Mg/nzs+3Zmo7etd3pwS/v1FUtu4w6uT2E5m+/mNml76MtT4 dDbg==
X-Gm-Message-State: AIVw1135eLwHRmOU3ITRaRzX/n7ns41L/kFddz6EKmmjqcXe4S9dcRk4 /uXkoVcI+qx4EcWq8ODHHD7/qvujbA==
X-Received: by with SMTP id m68mr18497038ith.16.1501526824952; Mon, 31 Jul 2017 11:47:04 -0700 (PDT)
MIME-Version: 1.0
Received: by with HTTP; Mon, 31 Jul 2017 11:47:04 -0700 (PDT)
From: David Noveck <davenoveck@gmail.com>
Date: Mon, 31 Jul 2017 14:47:04 -0400
Message-ID: <CADaq8jf8s+qpgC25d75Jm4=Mk9mcb5=TEYpKR9AzkczgV7dwag@mail.gmail.com>
To: Chuck Lever <chuck.lever@oracle.com>, "nfsv4@ietf.org" <nfsv4@ietf.org>
Content-Type: multipart/alternative; boundary="94eb2c11ce4a05aa980555a17522"
Archived-At: <https://mailarchive.ietf.org/arch/msg/nfsv4/LUFCZh9FqW8cEerE5Sg_F4EKsYg>
Subject: [nfsv4] Credit management and one-way messages
X-BeenThere: nfsv4@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: NFSv4 Working Group <nfsv4.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/nfsv4>, <mailto:nfsv4-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nfsv4/>
List-Post: <mailto:nfsv4@ietf.org>
List-Help: <mailto:nfsv4-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/nfsv4>, <mailto:nfsv4-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 31 Jul 2017 18:47:08 -0000

One issue that has been mentioned as a possible impediment to advnacing
RPC-over-RDMA to be a working group document, in its current form, concerns
its use of one-way messages to convey transport properties.  These concerns
seem to arise from the presentation of credit managemt as a "request-grant
protocol".  If the credit logic in fact depended on that pairing, then
there would be no roon for one-way meesages.

In fact, it doesn't. If you look at the actual operation of credit
management, one sees a different picture from what is stated in the
introductory paragraph of Section 3.1.1 of RFC 8166:

Flow control for RDMA Send operations directed to the Responder is implemented
as a simple request/grant protocol in the RPC-over-RDMA header associated
with each RPC message.

what follows focuses on the role of the grant. giving extensive information
but how it is computed and nd why it is vsli for the requester to use it.

While the text says "Practically speaking, the critical value is the
granted value", it is not clear what the exact role of the request is.  It
aapears to be a hintwhich the receiver is under no obligation to take any
notice of.  So it appears that the presentation of the creddit management
approach as a request-grant protocol., is a reflection of the fact that,
when it is used to transport RPC Requests and Replies, this pairing always
exists.  When grants are sent with one-way messages, there is no problem
that arises from the fact that there is no corresponding credit request.
The sender simply informs the receiver about his own receive resources and
thus his ability to accept further sends.

There is one potential issue connected with use of one-way messages to be
addressed.  If the recever has N credits and then N one-way messages are
sent without any traffic in the opposite direction, then it is possble for
a deadlock to result, since there would be no way for the sender to find
out about the receive resources.  For most one-way messages, this is not a
problem, since many one-way messages naturally give rise to messages in the
opposite direction even if the relationship is not formalized witin an RPC
paradigm.  For example:

   - The RDMA2_CONNPROP sent by the client to the server is paired with an
   RDMA2_CONNPROP in the opposite direction.

   An RDMA2_REQPROP results in RDMA2_RESPROP send in response.

RDMA2_UPDPROP is an exception. In the unlikely event that there are a a
lage series of such messages sent in one irection while there are no RPC's
being sent to the receiver it is possible for a ealock to arise. This would
result in a situation in which the receiver would have to send some message
back in the other to provide a credit grant. The most likely way to to that
is for it to choose to send an RDMA2_UPDPROP with an empty property set in
the reverse direction.

One other problem with one-way messages as they stand in rpcrdma-version-two
concerns section 7.2.2 in which the following item in the bullet list is

   - When the rdma_proc field has the value RDMA2_OPTIONAL and no RPC
message payload is present, a Requester MUST set the value of the
rdma_optdir field to CALL, and a Responder MUST set the value of the
rdma_optdir field to REPLY.  The Requester chooses a value for the
rdma_xid field from the XID space that matches the message's
direction.  Requesters and Responders set the rdma_credit field in a
similar fashion: a value is set that is appropriate for the direction
of the message.

This cannot be acted on, because, in the context of a one-way message,
it is not clear which party is the requester and which is the
responder.  While the roles of client and server are fixed and clear,
the roles of requester and responder vary from RPC to RPC.  If you are
not in an RPC context, then any decision as to who is the requestor or
responder is arbitrary.

I think the best way to address these issues is for rpcrdma-version-two
would be to:

   - Provide an explanation of credit management not so tied to the RPC
   - Add a new value for the direction field for one-way messages
   - Provide that one-way messages always contain a credit grant rather
   than a creit request
   - Explan how the potential deadlock with RDMA2_UPDPROP can be avoided.