Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

Rick Macklem <> Tue, 16 April 2024 02:31 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id C8FBAC14CF1A for <>; Mon, 15 Apr 2024 19:31:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id dTI8k2bbwaBn for <>; Mon, 15 Apr 2024 19:31:21 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::102b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by (Postfix) with ESMTPS id 5F92FC14F701 for <>; Mon, 15 Apr 2024 19:31:21 -0700 (PDT)
Received: by with SMTP id 98e67ed59e1d1-2a614b0391dso2586057a91.1 for <>; Mon, 15 Apr 2024 19:31:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20230601; t=1713234681; x=1713839481;; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ZYFXHBoPBQVRiZRoIBMsqZYco5VQW8uL94WyeLI9GRI=; b=N7I3SIC7kIpfgpsyDqAdgo4tEi7lrV48nW8vMIe6r5z7/SHmeK3E9wg/ZdD+9ZlIWs I4hc1xPwUQ6Q6Tops8kU5407Httyvw4VVtxmyr8d9/ru6IZ3Hch6a6jNTGb4P6xwGbpJ nI4mqJSOGKjicctF44j83138X1jwLHiR4f246RgqAcZrNZBOPjdDBODYB7Cj7K20sRSF AFVD/0872iOg5kCwmESfWiLznBRGvIMvqnO1ZK8NWRxd94E/TPD8U4mSxqPJMwAf7JVu JqopVgtictRTUJ9VyRoHFPwlaDKObSIvDAnuhLh4zXB2qfQPZ/epXMphWuNKV7SWVbB0 4imw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20230601; t=1713234681; x=1713839481; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZYFXHBoPBQVRiZRoIBMsqZYco5VQW8uL94WyeLI9GRI=; b=OhLXi9isCvfzY8gZA2XlJa4h4e1jj4QlTtkanf09ISZ0l/p0L/NDxF4RMsw9993j9K rs7Ac/258AU8eZ1BFw8hbzY41NUdEJdHI9E1jtHkewKF7x7g2VthNMB9NJAD5fUzBiDJ F674I3J37f0XAKxSPuRwYS8t017tkMTAoyVkCiFPVlXe3xzX5nZDbAN2i86+zZetQk/B BWjLOBVoBxjnT0E14o+qhndI5LbkCUGTWPK2jsZQgarqdZkw1uJhoDWg3uNNynI1/C1R 8psfRdlKnJGtAhBVzr9RtX1pEBIyXpRbW4NMPhjsolaMM6wy2stUGyhTN/tHZi5Gj3Mp Ce3A==
X-Gm-Message-State: AOJu0YyTeMs+5PjMLFKm4fFsEbQDa1BIcNmqaFUF1TZgt7YAVjLea+cx GH6f43QlBPTa9WdIUjpmy1nJ8zLgLp9ttLrcmfcbPsdzN4Xn7JMfFvQVvCAzzToDyDm3s67HgIu 4Yb3nLns3UqvGsBkGGzmx/jb6W3De
X-Google-Smtp-Source: AGHT+IF/TqvOk3HDo2oxI3a3+3OLzin+ieB5GLLjDUOBfs+RsCSBevxJfvAyaXWSDdizuF4Qtj+UsCvAoLqC32/Bp5A=
X-Received: by 2002:a17:90a:de8e:b0:2a2:a243:478f with SMTP id n14-20020a17090ade8e00b002a2a243478fmr9313490pjv.1.1713234680645; Mon, 15 Apr 2024 19:31:20 -0700 (PDT)
MIME-Version: 1.0
References: <> <>
In-Reply-To: <>
From: Rick Macklem <>
Date: Mon, 15 Apr 2024 19:31:09 -0700
Message-ID: <>
To: Trond Myklebust <>
Cc: "" <>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <>
Subject: Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: NFSv4 Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Tue, 16 Apr 2024 02:31:21 -0000

On Mon, Apr 15, 2024 at 5:05 PM Trond Myklebust <> wrote:
> On Apr 15, 2024, at 18:31, Rick Macklem <> wrote:
> Hi,
> I am wondering if others have found this a pita and are interested in
> a new operation for NFSv4.2 to get around it.
> Right now DelegReturn requires a CFH and delegation stateid.
> I do not know why the CFH is needed, since it seems that the
> delegation stateid should be sufficient?
> This new operation would be the same as DelegReturn except that
> it would not require a CFH argument.
> Why am I interested in this?
> - Lets assume a NFSv4.2 server that supports persistent FHs and
>  multiple hard links (as most extant servers do).
> - A client mounted to this server is removing a file for which it holds
>  a write delegation and a bunch of dirty data.
> Right now, the client must:
> - Write the dirty data back to the server.
> - DelegReturn
> - Remove
> because the client has no way of knowing if the file will be deleted
> by the Remove (or alternately is just removing one of several hard links).
> If a client is holding a delegation, it will presumably have done a GETATTR at some point while holding that delegation so should know how many links remain. While it is true that the unlink might invalidate the filehandle, that’s not a situation you are able to fix with the new operation.
> With a DelegReturn_NoFH, the client could:
> - Do a compound RPC with
>  Remove
>  PutFH of file
> - if the above returns NFS_OK, the file still exists and the client
>  can continue to use the delegation, writing the dirty data back
>  to the server sometime later (or maybe never, if the file gets
>   deleted by a subsequent Remove).
> or
> - If the above returns NFS4ERR_STALE for the PutFH, the file
>  has been deleted. It can throw the dirty data away and
>  DelegReturn_NoFH the delegation.
> A similar situation exists for Rename where the destination
> already exists.
> I find this scenario happens frequently during software builds.
> So, what do others think  w.r.t. this new operation? rick
> Why would you need to do a DELEGRETURN if the PUTFH returns NFS4ERR_STALE?
> If the server isn’t willing to just invalidate the delegation on the final unlink of the file, then it can and should recall the delegation (and any pNFS layouts that might be outstanding).
Good point. I thought that doing a DelegReturn was always required to be done by
a client, but I do not see anything in the RFC that states that. It
notes that a DelegReturn
can be done either before or when a CB_RECALL occurs. It does not seem to say
anything about the case where the file is being deleted on the server.

If it is fair to assume the server with either CB_RECALL or just
discard the delegation
when a file is deleted, that seems reasonable.
It is obvious that a CB_RECALL is required if the REMOVE is done by a different
client than the one holding the delegation. For the case where the delegation is
held by the same client as doing the REMOVE, I had assumed the CB_RECALL
was not required, but I am not sure that is what the RFC says (see the
other reply).

Anyhow, if the client does not need to DelegReturn except for a CB_RECALL,
then this operation is not needed because the client can try to do the PutFH
after the REMOVE and if it fails, just not do anything more.
(If the server cared, it would do a CB_RECALL as a part of the REMOVE.)

So, basically, forget about this suggested new operation, rick

> _________________________________
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace