Re: [nfsv4] NFS over QUIC

Chuck Lever <> Thu, 03 September 2020 23:48 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 440AF3A1398 for <>; Thu, 3 Sep 2020 16:48:24 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id oN5nXREm8Xiq for <>; Thu, 3 Sep 2020 16:48:22 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::832]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 744D93A1397 for <>; Thu, 3 Sep 2020 16:48:22 -0700 (PDT)
Received: by with SMTP id 19so1893367qtp.1 for <>; Thu, 03 Sep 2020 16:48:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=lwfq6FpRJclrB1/WzyzhGXyWxqVY2r+obdBZ8s9OSrA=; b=UqEDZU+nV8uXVgJ5TSjf/S7LqNd+pjxKPMWrc1Z3iOGEOYHhsU0l0AYF1dM+pi11jU nMutSXg5nP+YKoaM+NVNP0lOF3Xw5Ecbfw/10iHnVmBkZGqsfNmDRfL53jPRayU0Z4bJ kX0RFJuV7Qo5krbN4QM1HZWphwT5rUzkr6Wj5dcAZYzsbB9MxaAn1qbzCqREnDUx3p4n 9nk++rLD0RWvYquC8pKutd6psctL0QOUsGxBGxCJz9z/gZW2N1tIKnH94/h81Eu3tajD QSSSlRLgxXrSTySNiwscnD6LD+K8RNWjzmGTCPcxU3MMORxaj3o7K/7961GHSUAHKjZm jveA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=lwfq6FpRJclrB1/WzyzhGXyWxqVY2r+obdBZ8s9OSrA=; b=DrjyUQ2mavEMJ2N0m35JLnEtGI4nVjzaOOljgGY+hcrgfnyLtojHeeblEWW0UuPMhe vwJWpHPuz18qjONXk/a3D5NzaftfKrp1bRlRbjlhn6G3I4ugfk3FL4ZIcsiLFI72nR/q COI3f6QeAEaqr3EEJm46qsOQvwY/PgyUDhcD8mY6SHrycIblYCYWYYyGA+vVpyWc8pLM nkXxLUQr64aLcaNpMQPe9vkcL5kr9vddMT9ZbhzkLgyjuYKgfuFJmBLOg0mVOncxZyeu wzZ9qXPsdHFnB2Q2wxykEbCCGhRsHP9Ffo1OGKhtzLMfUDWsIAUNcx7owGLQvzaKMlo9 nHOg==
X-Gm-Message-State: AOAM533xCMfAcW/X8gANwHOaMTE65z3KPDkPyRgY6WpVy9vMNWaHDVZo rMcv03zgWIhD56kYqOWQ3Y0=
X-Google-Smtp-Source: ABdhPJyCuYgUuEdb0SNkTUQPrSNwFL+i0RdiMO0VXhjXv1MrHRYY7C0wivRw2iR7i3ak6cZlnDAmqA==
X-Received: by 2002:ac8:6b53:: with SMTP id x19mr6048113qts.296.1599176901379; Thu, 03 Sep 2020 16:48:21 -0700 (PDT)
Received: from ( []) by with ESMTPSA id z37sm3400241qtz.67.2020. (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Sep 2020 16:48:20 -0700 (PDT)
Content-Type: text/plain; charset="us-ascii"
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.\))
From: Chuck Lever <>
In-Reply-To: <>
Date: Thu, 03 Sep 2020 19:48:19 -0400
Cc: Linux NFS Mailing List <>,,
Content-Transfer-Encoding: 7bit
Message-Id: <>
References: <>
To: Bruce Fields <>
X-Mailer: Apple Mail (2.3608.
Archived-At: <>
Subject: Re: [nfsv4] NFS over QUIC
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: NFSv4 Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 03 Sep 2020 23:48:24 -0000

Hi Bruce-

> On Sep 3, 2020, at 5:52 PM, J. Bruce Fields <> wrote:
> I've been thinking about what might be required for NFS to run over
> Also cc'ing Steve French in case he's thought about this for CIFS/SMB.
> I don't have real plans.  For Linux, I don't even know if there's a
> kernel QUIC implementation planned yet.
> QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft:

The link to the latest version of that document is

> For example, section 4.3, which explains how to authenticate on top of
> an already-encrypted session, should also apply to QUIC.

Most of the document's content will be re-used for defining
RPC-over-QUIC, for example the ALPN defined in Section 8.2.
Lars Eggert, a chair of the QUIC WG, has been helping guide
the RPC-over-TLS effort with an eye towards using QUIC for
RPC when QUIC becomes more mature.

I thought the plan was to write a specification of RPC-over-
QUIC as a new RPC transport type with a netid and uaddr along
with a definition of the transport semantics (a la TI-RPC).
The document would need to explain record marking, peer
authentication, how to use multi-path and multi-stream support,
and so on.

Making NFS work on that transport should then be straightforward
enough that perhaps additional standards work wouldn't be

> QUIC runs over UDP, so I think all that would be required to negotiate
> support would be to attempt a QUIC connection to port 2049.
> The "Transport Layers" section in the NFS RFCs:
> requires transports support reliable and in-order transmission, forbids
> clients from retrying a request unless a connection is lost, and forbids
> servers from dropping a request without closing a connection.  I'm still
> vague on how those requirements interact with QUIC's connection
> management and 0-RTT reconnection.
> looks
> useful, as a guide for applications running over QUIC.  It warns that
> connections can time out fairly quickly.  For timely callbacks over NFS
> sessions, that means we need the client to ping the server regularly.
> Sounds like that's what they do for HTTP/QUIC to make server push
> notifications work:
> 	HTTP clients are expected to use QUIC PING frames to keep
> 	connections open.  Servers SHOULD NOT use PING frames to keep a
> 	connection open.  A client SHOULD NOT use PING frames for this
> 	purpose unless there are responses outstanding for requests or
> 	server pushes.
> QUIC allows multiple streams per connection--I wonder how we might use
> that.  RFC 5661 justifies the requirement for an ordered transport with:
> 	Ordered delivery simplifies detection of transmit errors, and
> 	simplifies the sending of arbitrary sized requests and responses
> 	via the record marking protocol.
> So as long as we don't try to split a single RPC among streams, I think
> we're OK.  Would a stream per session slot be reasonable?  I'm not sure
> what the cost of a stream is.
> Do we need to add a new universal address type so the protocol can
> specify QUIC endpoints when necessary?  (For server-to-server-copy, pnfs
> file layouts, fs_locations, etc.)  All QUIC needs is an IP address and
> maybe a port, so maybe the existing UDP/TCP addresses are enough?

Chuck Lever