Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

David Noveck <> Tue, 16 April 2024 12:21 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id E608EC14F712 for <>; Tue, 16 Apr 2024 05:21:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -7.095
X-Spam-Status: No, score=-7.095 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id KItHD8Y8YrMD for <>; Tue, 16 Apr 2024 05:21:18 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::c2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by (Postfix) with ESMTPS id 1310EC14F6AC for <>; Tue, 16 Apr 2024 05:21:18 -0700 (PDT)
Received: by with SMTP id 006d021491bc7-5aa318db8a0so2821795eaf.1 for <>; Tue, 16 Apr 2024 05:21:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20230601; t=1713270077; x=1713874877;; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=J4ws4aIdKsRC0oXBqIm2E8/h4jJxy0inPDIxaHZzlo8=; b=jk/aDIWX7DaBJy+mfUUofnHtytBMZdDiqx69ygy39FdmfKWYazAAtL22KOWHf655GR 1zHpWT4UPFN2JRhKsFI+QTQeaJPQiLb3CohoOUQqG7NV2HZQEeueq0lvYxLE30xaGo1Q nVQXh8P2w03iWZjahuY6Pb0k92sRF3k9NQQE5iUfl6c8C+GgurZRQwHwTgbI3eBEdhE4 OpWp22yMBZfRjSvYeTQDV/7Vwz2JhKw0oFhINCeKQVCCUK1OJeUpdyVShwFMFptBZdQj RBVTTdcycMd8kMOXu6VYxL6u+onDG4m5SgPh1jRZ7AVURczpKKiF+r+R8CEOWUs7cU59 Ao8Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20230601; t=1713270077; x=1713874877; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=J4ws4aIdKsRC0oXBqIm2E8/h4jJxy0inPDIxaHZzlo8=; b=CwN/37IC4SqH3kz4fppRjkKyYBnkWjsmOz94r95uIIRDk0JaG+EhyB7BNqPlZu/Z/J G81ZPW7IVwOSnfoVVV5ZD3ZdBxvJIZuppvyPz279AxHVakL9rftg7ziDQxGtwXa1EhZs 18h8YKu3mLnPniootHsYIRTf0mMlkhBckNxXrES3KWhkLt6qWP3wCp5YYN+/C9EUIAdS CghS2QCtS0LxIhpGBiEx4kVdApYD4Oqnb8PuBnKpAECueApvVUZo6T7JG4D2tM4PaSFL d0nSiHgetRqnL/UaQImpnA+S408zYxle+AnLSQiezMHAyp9amOj90o/wRQFrGs7ZGBa+ acXQ==
X-Forwarded-Encrypted: i=1; AJvYcCWT1IpDuWDWawsFpCDsAipHBLQd1VjuQvQbkc3AMqdCZskGpfWWPqAWxUrWJwO7qUIHAsWfQHpw3GyTEFa7kA==
X-Gm-Message-State: AOJu0YwQ6VkDFGTp5g4fXIBtYjpMIvtCQPrfajEmoc/2OYoQYITprz5P kvKwhLBKxqDc2R34XuDYBFpecfWI+cnkhMH8TVeXI53/1uDQ0NcRPiFhFVI6EguFdZVv+2jUTy0 Jv3VRtfqv0gRJN8aSQDXVoCgA4LUm7g==
X-Google-Smtp-Source: AGHT+IHyI6hZMQM4QO1L/gvziCSCQDnZNCucOPC4/JpefDiH5AtQTSEGllSAQfgypCWfck8YrsScFfVRnn9SFMCpQe4=
X-Received: by 2002:a05:6358:228c:b0:186:1146:2385 with SMTP id t12-20020a056358228c00b0018611462385mr12533222rwb.8.1713270077151; Tue, 16 Apr 2024 05:21:17 -0700 (PDT)
MIME-Version: 1.0
References: <> <>
In-Reply-To: <>
From: David Noveck <>
Date: Tue, 16 Apr 2024 08:21:04 -0400
Message-ID: <>
To: Trond Myklebust <>
Cc: Rick Macklem <>, NFSv4 <>
Content-Type: multipart/alternative; boundary="0000000000005a6406061635c66a"
Archived-At: <>
Subject: Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: NFSv4 Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Tue, 16 Apr 2024 12:21:22 -0000

On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust <>

> On Apr 15, 2024, at 18:31, Rick Macklem <> wrote:
> Hi,
> I am wondering if others have found this a pita and are interested in
> a new operation for NFSv4.2 to get around it.
> Right now DelegReturn requires a CFH and delegation stateid.
> I do not know why the CFH is needed, since it seems that the
> delegation stateid should be sufficient?
> This new operation would be the same as DelegReturn except that
> it would not require a CFH argument.
> Why am I interested in this?
> - Lets assume a NFSv4.2 server that supports persistent FHs and
>  multiple hard links (as most extant servers do).
> - A client mounted to this server is removing a file for which it holds
>  a write delegation and a bunch of dirty data.
> Right now, the client must:
> - Write the dirty data back to the server.
> - DelegReturn
> - Remove
> because the client has no way of knowing if the file will be deleted
> by the Remove (or alternately is just removing one of several hard links).
> If a client is holding a delegation, it will presumably have done a
> GETATTR at some point while holding that delegation so should know how many
> links remain. While it is true that the unlink might invalidate the
> filehandle, that’s not a situation you are able to fix with the new
> operation.
> With a DelegReturn_NoFH, the client could:
> - Do a compound RPC with
>  Remove
>  PutFH of file
> - if the above returns NFS_OK, the file still exists and the client
>  can continue to use the delegation, writing the dirty data back
>  to the server sometime later (or maybe never, if the file gets
>   deleted by a subsequent Remove).
> or
> - If the above returns NFS4ERR_STALE for the PutFH, the file
>  has been deleted. It can throw the dirty data away and
>  DelegReturn_NoFH the delegation.
> A similar situation exists for Rename where the destination
> already exists.
> I find this scenario happens frequently during software builds.
> So, what do others think  w.r.t. this new operation? rick
> Why would you need to do a DELEGRETURN if the PUTFH returns NFS4ERR_STALE?

To make sure the client and server agree on the state of things.

> If the server isn’t willing to just invalidate the delegation on the
final unlink of the file,

I'm not sure about "unwilling",  but I am uncomfortable with unilateral
stare changes like this, even if the spec indicated that needed to be done,
which is harder to effect than an extension such as Rick is proposing

 > then it can and should recall the delegation (and any pNFS layouts that
might be outstanding).

If it did recall it/them, then there is no way to return them without
something like Rick's proposed extension.

It seems to me we need something like CB_REVOKED (aka ?
CB_RECALL_WITH_EXTREME_PREJUDICE :-) or some equivalent that might be dealt
with in rfc5661bis-05

I think we should reserve 5-10 minutes of the 4/23 interim to discuss
possible approaches to dealing with the problem of getting rid of locking
state for removed files.

> _________________________________
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> _______________________________________________
> nfsv4 mailing list