Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

Rick Macklem <> Tue, 16 April 2024 14:06 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id C5739C14CF1E for <>; Tue, 16 Apr 2024 07:06:39 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -7.096
X-Spam-Status: No, score=-7.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id zgzeRtRrcpBT for <>; Tue, 16 Apr 2024 07:06:35 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::431]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by (Postfix) with ESMTPS id A63C2C14F6AE for <>; Tue, 16 Apr 2024 07:06:27 -0700 (PDT)
Received: by with SMTP id d2e1a72fcca58-6edc61d0ff6so4064797b3a.2 for <>; Tue, 16 Apr 2024 07:06:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20230601; t=1713276387; x=1713881187;; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=cNaFt10kd1jgTe5QVY+1KrG6btz57QOsOksSp5a09uI=; b=T5QoZ6Tz0RSHl7VgUEMOPlWqhMEjnmoXCQtRMk+wtFInF+3BqLMaD0JqrsUPl48mtT I20hEG9s6qiiM5zNxMPNenrh/ds9PcOnVMIYKS2+i8+4TJPG/w1I+Uhh/VQeVacRfh1M ls/6UXxFH4jVpVVd8/XkrMhma8IRT4HRQ2czO41A1qH47saK9RjBWFVAg1af35PGIhNL 4fR9FcCo7Kxxeuq4FPHpbf5oxVA8Ff364056g6P+JH/OyUEXtnTU40xPKcQzF62MtEdC aaxRDGuScRoqR6sMRWor/Q66jTHFDTsN2YjEF5HcS2sHiD6KnuoKpaQr/uuhlwtQa0iP VKpA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20230601; t=1713276387; x=1713881187; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cNaFt10kd1jgTe5QVY+1KrG6btz57QOsOksSp5a09uI=; b=NqV56Dh59kWq9vuFnI8FxBwBWcsO7njZHwYwEitm+JI7Z6C5YNk8Ia7uzFRSUobtpV 0kwIP0X+pum6DTro9MvBpVkvFqgnX+g2l21oiFJelrjv4GO6+JZuqEOqv4lY3YA88YH7 zzGoC0/HLEy+rk2KpdlVGUrl0eJDK53DFkhI84O+//C4oB9Y7uSOCyL9hY8ciq0+jBho uB/lLnH1FqzFGCmEmGG3ET4iCFVbjh1OjLF3hOP5tZoUH5XRS5wvnNw/afW2Bj87o2FN 18KxGXzyUd5zhQnDEpty9mdVzqdHASlAfn7P86Xbvb3AscLABIVzWtNozyWiWqTufnLI bD/A==
X-Forwarded-Encrypted: i=1; AJvYcCVfxcXRMs1r7S5cdtXAwl+SVvgrPWudSw59NnR7pTfWVvApjG45CUIVzT9xkg59sbZPivtoBuWchTBKou8IQg==
X-Gm-Message-State: AOJu0YxFDD5D/CwkLNy6UKFdBwwAnx3Dy+FU4Au2pmQMyqpIvt1VlXPT /0fAm4g9YluQw7gElPa4E9rH5trJ+5domVLEl5bjjcpTgA7+2FHal1cnI8QgR0jSkCKzGNn/BlE O74QtXYeoSdNF5Jup5b/4/EyxSA==
X-Google-Smtp-Source: AGHT+IGMCdBHNR2Z7QSoM7D/8c6QCuyDO6SI7OuqeBEgjqdkxRfJlBPw+J9cpDyYQByCEgP3tdJmMN3pX1r1pxgh6ws=
X-Received: by 2002:a05:6a20:3516:b0:1a9:7f1b:e3f7 with SMTP id d22-20020a056a20351600b001a97f1be3f7mr13645697pze.50.1713276386549; Tue, 16 Apr 2024 07:06:26 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <>
In-Reply-To: <>
From: Rick Macklem <>
Date: Tue, 16 Apr 2024 07:06:15 -0700
Message-ID: <>
To: David Noveck <>
Cc: Trond Myklebust <>, NFSv4 <>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <>
Subject: Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: NFSv4 Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Tue, 16 Apr 2024 14:06:39 -0000

On Tue, Apr 16, 2024 at 5:21 AM David Noveck <> wrote:
> On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust <> wrote:
>> On Apr 15, 2024, at 18:31, Rick Macklem <> wrote:
>> Hi,
>> I am wondering if others have found this a pita and are interested in
>> a new operation for NFSv4.2 to get around it.
>> Right now DelegReturn requires a CFH and delegation stateid.
>> I do not know why the CFH is needed, since it seems that the
>> delegation stateid should be sufficient?
>> This new operation would be the same as DelegReturn except that
>> it would not require a CFH argument.
>> Why am I interested in this?
>> - Lets assume a NFSv4.2 server that supports persistent FHs and
>>  multiple hard links (as most extant servers do).
>> - A client mounted to this server is removing a file for which it holds
>>  a write delegation and a bunch of dirty data.
>> Right now, the client must:
>> - Write the dirty data back to the server.
>> - DelegReturn
>> - Remove
>> because the client has no way of knowing if the file will be deleted
>> by the Remove (or alternately is just removing one of several hard links).
>> If a client is holding a delegation, it will presumably have done a GETATTR at some point while holding that delegation so should know how many links remain. While it is true that the unlink might invalidate the filehandle, that’s not a situation you are able to fix with the new operation.
>> With a DelegReturn_NoFH, the client could:
>> - Do a compound RPC with
>>  Remove
>>  PutFH of file
>> - if the above returns NFS_OK, the file still exists and the client
>>  can continue to use the delegation, writing the dirty data back
>>  to the server sometime later (or maybe never, if the file gets
>>   deleted by a subsequent Remove).
>> or
>> - If the above returns NFS4ERR_STALE for the PutFH, the file
>>  has been deleted. It can throw the dirty data away and
>>  DelegReturn_NoFH the delegation.
>> A similar situation exists for Rename where the destination
>> already exists.
>> I find this scenario happens frequently during software builds.
>> So, what do others think  w.r.t. this new operation? rick
>> Why would you need to do a DELEGRETURN if the PUTFH returns NFS4ERR_STALE?
> To make sure the client and server agree on the state of things.
> > If the server isn’t willing to just invalidate the delegation on the final unlink of the file,
> I'm not sure about "unwilling",  but I am uncomfortable with unilateral stare changes like this, even if the spec indicated that needed to be done, which is harder to effect than an extension such as Rick is proposing
>  > then it can and should recall the delegation (and any pNFS layouts that might be outstanding).
> If it did recall it/them, then there is no way to return them without something like Rick's proposed extension.
I assumed Trond meant "recall during the REMOVE" while the FH is still valid.


> It seems to me we need something like CB_REVOKED (aka ? CB_RECALL_WITH_EXTREME_PREJUDICE :-) or some equivalent that might be dealt with in rfc5661bis-05
> I think we should reserve 5-10 minutes of the 4/23 interim to discuss possible approaches to dealing with the problem of getting rid of locking state for removed files.
>> _________________________________
>> Trond Myklebust
>> Linux NFS client maintainer, Hammerspace
>> _______________________________________________
>> nfsv4 mailing list