Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

Trond Myklebust <trondmy@hammerspace.com> Fri, 19 April 2024 01:24 UTC

Return-Path: <trondmy@hammerspace.com>
X-Original-To: nfsv4@ietfa.amsl.com
Delivered-To: nfsv4@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A4504C15107A for <nfsv4@ietfa.amsl.com>; Thu, 18 Apr 2024 18:24:23 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level:
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_PASS=-0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=hammerspace.com
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mypbxiikwVLB for <nfsv4@ietfa.amsl.com>; Thu, 18 Apr 2024 18:24:19 -0700 (PDT)
Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2113.outbound.protection.outlook.com [40.107.102.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7F23EC151071 for <nfsv4@ietf.org>; Thu, 18 Apr 2024 18:24:19 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Aawkdp75CqCUId4th7MhDtYPhooDWyOV2bW9s4s43c0nYimHaOoJfbf6akU1XI1Z5ITsA4pX+HeMqlLtZZi+iF9ZuMCrcpzr9YONbn6ruTs8HgZt0Jn0JB/ZPuqJmyvqCeIPyZ1KmL2xkC1BUP2Aa4PC7pInbJM6/f/+2w0XRCuDXTwLV+F/qAH6ixKdfRZXXTGFFv2uvu/zoPwk5sAJ6D83/J4V0OH7+Dk5lEz2ailAq67Z4mYk35BJjGB7AeE/tGL9IKR0BQSPbgXPDeqt2YmS0iKqcvupquT3b2OJdtcLIqmKIYv8aDMgkbu1xwpJmbtiR04uWfHqkpsZqjRirw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UUJQjz9wqa57ME60pS53/IERxQN3paohQi6U+WHuEy8=; b=hv7mLfzzccSRHmT66dMUAFcXjxKPnkbgkLUIyf0Ea8QDSSNy2PgtmoVB+aE5lE5h8BmMcZ1WjZTKgXa/RYSzGPJfczfivbUBxuNrjW8wYxC69D0UixSfaQJznRU7r9phLx65jV81NWG3axVfEV12DHz8TLj1XAlVGWTbw78HGywogo4+5q0Q3GcN0B33dqFAs3Gjzp63Vu9Uz7hYj5+IOYmdxz3lJF7y24d274XBu7u3AHNtJrTL4gSJFKdS5ngSkPZqDJCHbl1n/qmCCgAXdujI+WhBr6ZAcBfWBdxpx+Lr9ZXBNoR0F6eTUsUwfCtCklLkZ+fHG2l742sahri8FA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=hammerspace.com; dmarc=pass action=none header.from=hammerspace.com; dkim=pass header.d=hammerspace.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hammerspace.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UUJQjz9wqa57ME60pS53/IERxQN3paohQi6U+WHuEy8=; b=bJ+pxeRSlAHHrnB5kVaaUbH9aZLz6510rrVNU37D6fLtjx6/1mAzs65d4OK2sdOJ9QJ8lLBLEFXqwnNErYcZFh+2AH6eSI1NgaxtGUjBA5OPNH4yrG/fSzelSF3S5Nj9oLKXn93wQxcrdq4Gu62BPngk+kUy/6IZAmlQkvxM680=
Received: from CH0PR13MB5084.namprd13.prod.outlook.com (2603:10b6:610:111::7) by BY1PR13MB6237.namprd13.prod.outlook.com (2603:10b6:a03:529::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7452.49; Fri, 19 Apr 2024 01:24:15 +0000
Received: from CH0PR13MB5084.namprd13.prod.outlook.com ([fe80::5bb5:501a:fb40:5057]) by CH0PR13MB5084.namprd13.prod.outlook.com ([fe80::5bb5:501a:fb40:5057%6]) with mapi id 15.20.7472.037; Fri, 19 Apr 2024 01:24:15 +0000
From: Trond Myklebust <trondmy@hammerspace.com>
To: "rick.macklem@gmail.com" <rick.macklem@gmail.com>
CC: "nfsv4@ietf.org" <nfsv4@ietf.org>
Thread-Topic: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?
Thread-Index: AQHaj5HPuHsSiF/StEqz/bB9rT0sn7Fq0fYAgAAdY4CAAAqNgIAAdGwAgADsOwCAAMwKgIAAC1qAgAAVnwCAAOpSAIAATB+AgAA5CACAAAaGgIAAE+EA
Date: Fri, 19 Apr 2024 01:24:15 +0000
Message-ID: <1ef5e112b29737b9a72bbc66d85fe8bb6c7b5f0f.camel@hammerspace.com>
References: <CAM5tNy407R30WQqCuzFE+feRz42wkaAAGOwiHwqFoWZSpwfT0A@mail.gmail.com> <9D17BB42-54A9-46DB-8E4F-3FF852EEAC90@hammerspace.com> <CADaq8jePTg8WVFe2FmrwgUjyc7H9v=Ln7g2FT9G5DJtkpVSF3Q@mail.gmail.com> <CAM5tNy6-ADHC1z9HJME0r5o-hDeG9OucE_SWftDrqE4JnMrhAg@mail.gmail.com> <adc3fb5ed3674078c64399b19891c63b7a1185e4.camel@hammerspace.com> <CAM5tNy77PNiH6aTWTOyhrofe2cFTg+ga8sFzGbi0EHSfrXz79w@mail.gmail.com> <CADaq8jdifYwzZDkrYnaZwV-_bYs=3z8HxfM7mtUAoPy29uvAig@mail.gmail.com> <d4b197a4dcad15d30b4aec25adbd5bb08485f121.camel@hammerspace.com> <CAM5tNy7cDpMd+zms1r6PjtVvZQ7LWyyrFmL6Pn_x1FkJYfPckA@mail.gmail.com> <CAM5tNy6vX6XUfwGLuOWOHWHBxhrL-Kr6Xe-OJT8hspZbuO6PRQ@mail.gmail.com> <6513a671fae58f522224fb608dd9735c2bb5ddaa.camel@hammerspace.com> <CAM5tNy4izPiqDCxJXkWCUCpjFRHx3uR9Po+i_m8O0pmFUFuLbw@mail.gmail.com> <d2f335cfe57f19a57e13f3304fdbe806e1bb3d5d.camel@hammerspace.com> <CAM5tNy7zZ8edF5KCoTJFjfndKyet1kaepcUbKV=EqquAR0_Dag@mail.gmail.com>
In-Reply-To: <CAM5tNy7zZ8edF5KCoTJFjfndKyet1kaepcUbKV=EqquAR0_Dag@mail.gmail.com>
Accept-Language: en-US, en-GB
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=hammerspace.com;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: CH0PR13MB5084:EE_|BY1PR13MB6237:EE_
x-ms-office365-filtering-correlation-id: a9154f1b-5173-49b7-6a3a-08dc600f6d70
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: q8kO/OLuw0RPknE7m2O+Pv1vDoTenQ1gTZbcBqdcttGaCZFOS0Hc9Aycmh4hvqH/mXZ1gi8JfUheP7reppqMmbjG4ZH+O79qH6FMI5UgMMBFNHbYoQzcgjm8yvnXU+ywJY8VTKl0iI1ShX0OCdJKtAsP+yqgn+Xm1z+//RkKHnxf8YBPmyB+x4Otz2UWa9I70i8WOiiCCK0AE8oFnKSqL7uNf8oHLFhhDMxGbjq1QIVrgISJVcyIREZHo/moeax0F3kVV2nv/QyT5mhkyfMYC0ecfj6YvHIFNWKo+2TeD5IedyM+7TnuWGYyCHR4zm9FanBzLcPn1wrUYxNzrBLr+DxU8JyhOJp9Fbcvhk3YBW9i/tO3cgy4PaTv4H9uQiK7fkBN16R2zk6hxbGZpbL/fQXKyi5PlyspGTjYe8kMlUKGD23OgUjPlwnzlsALsgqbU+SYJG2x/hpLrbhNUZv6gqjB96qoiNLzeYX8HlNMV6ON3yjJVYl8ZOgnRIh38l6ytofvPMaV7UZP7IdMlrImBo3v5A35JD5WJ02bdg6iUq3xZJoPZ9MieXxJxBSxQev8pqtd8dzACJcaHuUJ3TaVTMWN4es2MvuT7IXNVTuwhqmbts22ObawozkyXSwPC+5Jf1zYrA6GEODMGrv357XC4QBk9K7SMkJkCO6P9ZvC4XnILlDwcbrnSlDNeUUoRbeD2Y3AakrlJJbcI0djrBR39RUZLou4wTAyUS/MOiZXDsXbR43V3+4JcniZxrnC21uKw4uyMeuBoV52gpQ5xW8t4dxzzgGm3K030mj2f7CKKwFc7gLdKN+3BFBvzuhIkMu+4AQUV58/rXM5NgvXyJf5jTx5h7c1Zerb+GjFcWn7Wpxw2KiFV+M5ufektt1eenecZn9x8ejzUYTeYKsUmN8tZdOZ+2Hpr1KdfpRsKJpInucqlv0EPwf/9prpXS3rPG1cMjG2iYovXhPsRlN1u0D5ytQgBw3JG68bA7zQ50K9pPfO55nF0hqq8ZoVmWwsE4dcyJx/B0YmVt0vXO+jz9bQCBZHXR1j9ky+xaKhX8OV+FATyDfC+ZE/gQa9iRoo1ybZ8wMSFWGN+QG0g1RJhl1Ds9PBosJKjtpq2uS/wsjLKgKzH2+ByBZ4PKblVn/WPn+/mDc7MnEUbzXQFnn8ewl6VV12Vfk/8dfd0ZmwdbST4mPac4XYUk0EPjO0Tahi8eTx1zm+n/Zf0F3t9gAaBZappA5SszXyOht8Bc43P12gdEz5tnTf4b36fPy6LNHc3s7/7Shlcq9HZwR5C0vRrmkhFLSPynw19CvftikLI1axRQlllPe9+m6Gq2kwO9Xcxqc82ipvRUKNxqzecT5Vjc3eYg==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH0PR13MB5084.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(1800799015)(376005)(366007)(38070700009); DIR:OUT; SFP:1102;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: r/V2/a8C8RWsIfhnRAtIubGviCGaUNr4OBA5c7rLYNwPk0MugLBk4FXLJphg+tgSggRXdbJh1yaajiIUII/XJ+EiLdn7Lx9bn6vXA/RVVLnZqJXSIpWaMyCsjLeOV+2NhV8jN6Gk6m/YEzz1qaZsQQ3DMc/k8AYwL0pq+7vyakGiPvwlpGLXD4zv2u8mps4oiycD1p502LZV0vetbdahs4oo/zfDGOib3d0UIHWbAPSJN8qwgQ633FHMMkUsPyeyNZi0MIfWLJOy72aXobAoqlE7V+ncK9bHCh7R1oCHEA5aEfgn7WJuImkf9HSaubFbe/FpCKznMvpcu25PrvX7LhxtZ8NWadV2UqtMGapnHukBJXBYUH1+S4hi7Nd6p8pfZNURuBcnDLgqrxR3u/ve4h3mngb+uQa9wdqJLiEbIaeri3xlve5iyC0wTK3EJ46XNfy3PFxRUjtuTiRjzhlW/6a+vA7aLo4ZiaIRwjnlRJwJu03xOgrrF/38AKRqy6yWH3QWmu2ZxV4LxK15hqV6MmgjveZpcTYUpfLe3kLc4NEQMnX2cz8i9MCUDfVvRmrWFvyEyCsYgKRA+6ancE7m8L9xYdCX3Af876YjHm+2ZWQDDgFb/upBchGxtyuUue6gBaxtS+ndnxJpoOQAe3XXfHm7qGXvDP+qop8NjtYt/VrwhNq2RdnZmyD5Uym5cKhsbzUEfngEDDCT7WIIuOpTaH+mEf1NyxXeBApTzWUvcsq8SNrTbzYM/QZL6BmY8Kf6l7635Jiwrj/E5gbIAlE4TzUX6eQwvLET8qBXM3XPevtfBWxiz8nME17LBuCe7hJFD7N71PyujtDEzMJ2vwAZmOfJeuHBItrktk+mL/tm9g7+GXJyL+eWr9iqJPn/7dBLyFal8j8Zps6uuEJJaoJRoNPx5Qd8q1UH+ikQX4u7fc4vY/foD25RAGV02ornJ63cu4qKMgWTrzwKtPrnpzKTUEUFkpQ0g8vgNWYAyhqP09GU1ndbQo1AXri3uUrBnVuhB/afNJ3Kc86iEjFNd+kImxRlPmy2wJdCBSGj+Uaq1IlaA/bmLi7Gw/KwHyJVoVZb/W31OI7nIpua2WanRHCC+8XvXcVMBLt5qKpAAuz4K6OoLaE+Hk+X3R9SNH82CwWVkMNMLjQy90n7HGRMptp8eNELcXLyw8GL0tp4VkQf9fDbCE2ZRyoEl/3U0WrYzdvc+WEx9UoAWQOIHDJQbtk5Y1jh6+TItR/RyYjXwxpbwfMtQ++IOOv/yNk0g9gKmy4+zZamhSHVpOGp7Cubp01vgv+KIeSO3/uoDGBq1O3QB59GIiPXtcZhIrj368+lepPYXW37gloCE0hYugNhQTrN4YwaK/zisBFBsIZ51jhJr92pyVCfiIZq9kVFdYYHY7v+GopT8T7mGwObQ9JTApk0EpLtgqmmyIBlhD0crgS9nPLMZVyOp4XEAKTSC6viP6HnxnDE+3v87WXaLdmHek1r1W5vNBL0EN66LM/AulEFL0gFPhQqXjMTWEionx12T0Nj7GdFoH5ea4KbPwwZ29wv8nlivSAixUtf0KTQHD4L2aQvIA1N8LV/Z29aGXx9kWT/djqSPc1mY4U7nbqxkRCB5Lu1j9Qhg3y8LD01QrEGvOWTuQz7BbHjTQ6SChrFV6iFTfle9v9CZUCMODVy1LXIyg==
Content-Type: text/plain; charset="utf-8"
Content-ID: <A889122F8814AC4CAE3C407F0CCB0716@namprd13.prod.outlook.com>
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: hammerspace.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CH0PR13MB5084.namprd13.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a9154f1b-5173-49b7-6a3a-08dc600f6d70
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Apr 2024 01:24:15.6411 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 0d4fed5c-3a70-46fe-9430-ece41741f59e
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 5IiQYhSeaKJhiSdSVpSJN/D1xhIi0pUyaXTeHDnOo9C0E3Y6x6v6/X0EyoV83zXuT9i6S2gvzWXt3uwTBjcJUw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY1PR13MB6237
Archived-At: <https://mailarchive.ietf.org/arch/msg/nfsv4/mqdAWHws_332iRcfCr1OU5tRbxc>
Subject: Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?
X-BeenThere: nfsv4@ietf.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: NFSv4 Working Group <nfsv4.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/nfsv4>, <mailto:nfsv4-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nfsv4/>
List-Post: <mailto:nfsv4@ietf.org>
List-Help: <mailto:nfsv4-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/nfsv4>, <mailto:nfsv4-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 19 Apr 2024 01:24:23 -0000

On Thu, 2024-04-18 at 17:13 -0700, Rick Macklem wrote:
> On Thu, Apr 18, 2024 at 4:49 PM Trond Myklebust
> <trondmy@hammerspace.com> wrote:
> > 
> > On Thu, 2024-04-18 at 13:25 -0700, Rick Macklem wrote:
> > > On Thu, Apr 18, 2024 at 8:53 AM Trond Myklebust
> > > <trondmy@hammerspace.com> wrote:
> > > > 
> > > > On Wed, 2024-04-17 at 18:54 -0700, Rick Macklem wrote:
> > > > > On Wed, Apr 17, 2024 at 5:37 PM Rick Macklem
> > > > > <rick.macklem@gmail.com>
> > > > > wrote:
> > > > > > 
> > > > > > On Wed, Apr 17, 2024 at 4:56 PM Trond Myklebust
> > > > > > <trondmy@hammerspace.com> wrote:
> > > > > > > 
> > > > > > > On Wed, 2024-04-17 at 07:46 -0400, David Noveck wrote:
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > On Tue, Apr 16, 2024, 5:40 PM Rick Macklem
> > > > > > > <rick.macklem@gmail.com> wrote:
> > > > > > > 
> > > > > > > On Tue, Apr 16, 2024 at 7:44 AM Trond Myklebust
> > > > > > > <trondmy@hammerspace.com> wrote:
> > > > > > > > 
> > > > > > > > On Tue, 2024-04-16 at 07:06 -0700, Rick Macklem wrote:
> > > > > > > > > On Tue, Apr 16, 2024 at 5:21 AM David Noveck
> > > > > > > > > <davenoveck@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust
> > > > > > > > > > <trondmy@hammerspace.com> wrote:
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > On Apr 15, 2024, at 18:31, Rick Macklem
> > > > > > > > > > > <rick.macklem@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > 
> > > > > > > > > > > Hi,
> > > > > > > > > > > 
> > > > > > > > > > > I am wondering if others have found this a pita
> > > > > > > > > > > and
> > > > > > > > > > > are
> > > > > > > > > > > interested in
> > > > > > > > > > > a new operation for NFSv4.2 to get around it.
> > > > > > > > > > > 
> > > > > > > > > > > Right now DelegReturn requires a CFH and
> > > > > > > > > > > delegation
> > > > > > > > > > > stateid.
> > > > > > > > > > > I do not know why the CFH is needed, since it
> > > > > > > > > > > seems
> > > > > > > > > > > that
> > > > > > > > > > > the
> > > > > > > > > > > delegation stateid should be sufficient?
> > > > > > > > > > > 
> > > > > > > > > > > This new operation would be the same as
> > > > > > > > > > > DelegReturn
> > > > > > > > > > > except that
> > > > > > > > > > > it would not require a CFH argument.
> > > > > > > > > > > 
> > > > > > > > > > > Why am I interested in this?
> > > > > > > > > > > - Lets assume a NFSv4.2 server that supports
> > > > > > > > > > > persistent
> > > > > > > > > > > FHs and
> > > > > > > > > > >  multiple hard links (as most extant servers do).
> > > > > > > > > > > - A client mounted to this server is removing a
> > > > > > > > > > > file
> > > > > > > > > > > for
> > > > > > > > > > > which it
> > > > > > > > > > > holds
> > > > > > > > > > >  a write delegation and a bunch of dirty data.
> > > > > > > > > > > Right now, the client must:
> > > > > > > > > > > - Write the dirty data back to the server.
> > > > > > > > > > > - DelegReturn
> > > > > > > > > > > - Remove
> > > > > > > > > > > because the client has no way of knowing if the
> > > > > > > > > > > file
> > > > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > deleted
> > > > > > > > > > > by the Remove (or alternately is just removing
> > > > > > > > > > > one of
> > > > > > > > > > > several
> > > > > > > > > > > hard links).
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > If a client is holding a delegation, it will
> > > > > > > > > > > presumably
> > > > > > > > > > > have done
> > > > > > > > > > > a GETATTR at some point while holding that
> > > > > > > > > > > delegation
> > > > > > > > > > > so
> > > > > > > > > > > should
> > > > > > > > > > > know how many links remain. While it is true that
> > > > > > > > > > > the
> > > > > > > > > > > unlink
> > > > > > > > > > > might invalidate the filehandle, that’s not a
> > > > > > > > > > > situation
> > > > > > > > > > > you are
> > > > > > > > > > > able to fix with the new operation.
> > > > > > > > > > > 
> > > > > > > > > > > With a DelegReturn_NoFH, the client could:
> > > > > > > > > > > - Do a compound RPC with
> > > > > > > > > > >  Remove
> > > > > > > > > > >  PutFH of file
> > > > > > > > > > > - if the above returns NFS_OK, the file still
> > > > > > > > > > > exists
> > > > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > client
> > > > > > > > > > >  can continue to use the delegation, writing the
> > > > > > > > > > > dirty
> > > > > > > > > > > data back
> > > > > > > > > > >  to the server sometime later (or maybe never, if
> > > > > > > > > > > the
> > > > > > > > > > > file gets
> > > > > > > > > > >   deleted by a subsequent Remove).
> > > > > > > > > > > or
> > > > > > > > > > > - If the above returns NFS4ERR_STALE for the
> > > > > > > > > > > PutFH,
> > > > > > > > > > > the
> > > > > > > > > > > file
> > > > > > > > > > >  has been deleted. It can throw the dirty data
> > > > > > > > > > > away
> > > > > > > > > > > and
> > > > > > > > > > >  DelegReturn_NoFH the delegation.
> > > > > > > > > > > 
> > > > > > > > > > > A similar situation exists for Rename where the
> > > > > > > > > > > destination
> > > > > > > > > > > already exists.
> > > > > > > > > > > 
> > > > > > > > > > > I find this scenario happens frequently during
> > > > > > > > > > > software
> > > > > > > > > > > builds.
> > > > > > > > > > > 
> > > > > > > > > > > So, what do others think  w.r.t. this new
> > > > > > > > > > > operation?
> > > > > > > > > > > rick
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > Why would you need to do a DELEGRETURN if the
> > > > > > > > > > > PUTFH
> > > > > > > > > > > returns
> > > > > > > > > > > NFS4ERR_STALE?
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > To make sure the client and server agree on the
> > > > > > > > > > state
> > > > > > > > > > of
> > > > > > > > > > things.
> > > > > > > > > > 
> > > > > > > > > > > If the server isn’t willing to just invalidate
> > > > > > > > > > > the
> > > > > > > > > > > delegation on
> > > > > > > > > > > the final unlink of the file,
> > > > > > > > > > 
> > > > > > > > > > I'm not sure about "unwilling",  but I am
> > > > > > > > > > uncomfortable
> > > > > > > > > > with
> > > > > > > > > > unilateral stare changes like this, even if the
> > > > > > > > > > spec
> > > > > > > > > > indicated that
> > > > > > > > > > needed to be done, which is harder to effect than
> > > > > > > > > > an
> > > > > > > > > > extension such
> > > > > > > > > > as Rick is proposing
> > > > > > > > > > 
> > > > > > > > > >  > then it can and should recall the delegation
> > > > > > > > > > (and
> > > > > > > > > > any
> > > > > > > > > > pNFS
> > > > > > > > > > layouts that might be outstanding).
> > > > > > > > > > 
> > > > > > > > > > If it did recall it/them, then there is no way to
> > > > > > > > > > return
> > > > > > > > > > them
> > > > > > > > > > without something like Rick's proposed extension.
> > > > > > > > > I assumed Trond meant "recall during the REMOVE"
> > > > > > > > > while
> > > > > > > > > the FH
> > > > > > > > > is
> > > > > > > > > still valid.
> > > > > > > 
> > > > > > > 
> > > > > > > I didn't get that.  Apparently, I was mistaken.
> > > > > > > 
> > > > > > > > 
> > > > > > > > Yes. This is a clarification of how the server can deal
> > > > > > > > with
> > > > > > > > the
> > > > > > > > problem (if it thinks this is a problem) within the
> > > > > > > > confines of
> > > > > > > > the
> > > > > > > > existing spec
> > > > > > > 
> > > > > > > 
> > > > > > > That depends on what the problem is.  Your proposal does
> > > > > > > address
> > > > > > > the problem of recalling a delegation when it is
> > > > > > > incapable of
> > > > > > > being returned.  It doesn't, at least alone, solve Rick's
> > > > > > > problem  which is that he wants to be able to know, at
> > > > > > > the
> > > > > > > time
> > > > > > > of recall, whether he can throw away pending writes, as
> > > > > > > opposed
> > > > > > > to flushing them to the server.
> > > > > > > 
> > > > > > > 
> > > > > > > Please re-read what I said above (repeated here):
> > > > > > > 
> > > > > > > > > > > If a client is holding a delegation, it will
> > > > > > > > > > > presumably
> > > > > > > > > > > have done
> > > > > > > > > > > a GETATTR at some point while holding that
> > > > > > > > > > > delegation
> > > > > > > > > > > so
> > > > > > > > > > > should
> > > > > > > > > > > know how many links remain. While it is true that
> > > > > > > > > > > the
> > > > > > > > > > > unlink
> > > > > > > > > > > might invalidate the filehandle, that’s not a
> > > > > > > > > > > situation
> > > > > > > > > > > you are
> > > > > > > > > > > able to fix with the new operation.
> > > > > > I did not get this. Now I see that in LINK (sec. 18.9.3) it
> > > > > > says
> > > > > > that
> > > > > > delegations must
> > > > > > be CB_RECALL'd (I never noticed that and assumed that a
> > > > > > LINK
> > > > > > could
> > > > > > be done by another client when a delegation is held. My
> > > > > > Bad;-)
> > > > > > 
> > > > > > So, yes, the client that holds a delegation can track
> > > > > > numlinks
> > > > > > for
> > > > > > the
> > > > > > file. I didn't realize that was the case.
> > > > > I now think there is a glitch in this plan...
> > > > > I cannot see anywhere in RFC7530 that specifies LINK should
> > > > > recall
> > > > > delegations, so I am not sure if a NFSv4.0 LINK done by
> > > > > another
> > > > > client
> > > > > could increase numlinks without the delegation being
> > > > > recalled.
> > > > > (I knew there was a reason the FreeBSD NFSv4 server did not
> > > > > recall
> > > > > delegations for LINK. I will patch it to do so for
> > > > > NFSv4.1/4.2.)
> > > > > 
> > > > 
> > > > It doesn't just change nlink. It also bumps the change
> > > > attribute
> > > > (and
> > > > the ctime), which by itself implies that it must recall
> > > > delegations
> > > > according to RFC8881.
> > > > Specifically in section 10.2:
> > > > 
> > > >    *  The semantics of the file system are crucial in defining
> > > > when
> > > >       delegation recall is required.  If a particular change
> > > > within
> > > > a
> > > >       specific implementation causes change to a file
> > > > attribute,
> > > > then
> > > >       delegation recall is required, whether that operation has
> > > > been
> > > >       specifically listed as requiring delegation recall. 
> > > > Again,
> > > > what
> > > >       is critical is whether the guarantees provided by the
> > > > delegation
> > > >       are being invalidated.
> > > > 
> > > > The Linux client interprets that to mean that LINK will recall
> > > > delegations.
> > > Yes, I agree, for NFSv4.1/4.2.
> > > (And admit the FreeBSD NFSv4.1/4.2 needs to be fixed to do this.)
> > > 
> > > But what about NFSv4.0?
> > > I'm looking, but cannot find similar words in RFC7530.
> > > (Sec. 10.4.4. deals with Recall  of an Open Delegation, but it
> > > does not mention LINK and it does not mention "any operation
> > > that modifies the attributes. The same words appear to be in
> > > RFC3530, except under Sec. 9.4.4.) I do find mention that
> > > holders of delegations can either not modify attributes or
> > > only the attributes related to writing for write delegation.
> > > However, I cannot see any mention of clients that do not
> > > hold delegations.
> > > 
> > > Maybe you are arguing that it should have been obvious for
> > > NFSv4.0, but it was not so for me (and, therefore, maybe other
> > > NFSv4.0
> > > implementers?). There is also the case of NFSv3 LINK RPCs.
> > > 
> > > Although it would be nice to assume, I'm not convinced that
> > > extant NFS servers will all provide the guarantee that numlinks
> > > will
> > > not be changed (increased via LINK) when the client holds
> > > a delegation.
> > > (If the client could be sure the NFS server was a "pure
> > > NFSv4.1/4.2
> > > server and did not support earlier NFS versions, I would agree
> > > with
> > > you.)
> > > I just think that, since RFC7530 Sec. 10.4.4. makes no mention of
> > > LINK, it is risky to assume all NFSv4.0 server implementations
> > > recall
> > > delegations when a LINK is done. (In particular, since the
> > > NFSv4.0
> > > cannot know which client is performing the LINK.)
> > > 
> > 
> > What is the motivation for continuing to optimise NFSv4.0? While it
> > would be nice to allow it to benefit too, that problem is not one
> > that
> > keeps me awake at night.
> I certainly am not interested in improving NFSv4.0.
> 
> What I was trying to say is...
> When an NFSv4.0 client does a LINK, does the NFSv4.[0,1,2] server
> recall all delegations for the file?
> If not, then the numlinks attribute could get increased without the
> delegation being recalled. And if that is the case, then the
> NFSv4.1/4.2
> client cannot trust its value of numlinks to indicate if the file
> being
> REMOVE'd will be deleted on the server.
> 

From RFC7830, section 10.4:

   When a single client holds an OPEN_DELEGATE_READ delegation, it is
   assured that no other client may modify the contents or attributes of
   the file.  If more than one client holds an OPEN_DELEGATE_READ
   delegation, then the contents and attributes of that file are not
   allowed to change.  When a client has an OPEN_DELEGATE_WRITE
   delegation, it may modify the file data since no other client will be
   accessing the file's data.  The client holding an OPEN_DELEGATE_WRITE
   delegation may only affect file attributes that are intimately
   connected with the file data: size, time_modify, and change.

Clearly, when reading the above, the assumption should be that a read
delegation MUST be recalled or revoked if the file attributes change.

I do not find anything that says the same thing for a write delegation.
I suspect that the authors did intend to imply that, but I'm not
finding anything that explicitly says that a client that holds a write
delegation can expect attributes not to change due to the action of
another client. It can only expect that the data contents (and hence
the size attribute) won't change, and it is not allowed to modify
attributes other than those "intimately connected with the file
data"...

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com