Re: [netconf] restconf collections

"Per Andersson (perander)" <perander@cisco.com> Fri, 25 September 2020 12:31 UTC

Return-Path: <perander@cisco.com>
X-Original-To: netconf@ietfa.amsl.com
Delivered-To: netconf@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 567FE3A1053 for <netconf@ietfa.amsl.com>; Fri, 25 Sep 2020 05:31:24 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -9.62
X-Spam-Level:
X-Spam-Status: No, score=-9.62 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=cisco.com header.b=DqlwsF3A; dkim=pass (1024-bit key) header.d=cisco.onmicrosoft.com header.b=fkZaK+Lp
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qkneuyHX-dL7 for <netconf@ietfa.amsl.com>; Fri, 25 Sep 2020 05:31:22 -0700 (PDT)
Received: from alln-iport-2.cisco.com (alln-iport-2.cisco.com [173.37.142.89]) (using TLSv1.2 with cipher DHE-RSA-SEED-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8A1E13A0F53 for <netconf@ietf.org>; Fri, 25 Sep 2020 05:31:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=3978; q=dns/txt; s=iport; t=1601037082; x=1602246682; h=from:to:subject:date:message-id:references:in-reply-to: content-transfer-encoding:mime-version; bh=XDSU+q+gcPxKVj1bndHm1yKDs/Li+ovb6wB6VSNV1uU=; b=DqlwsF3Ak5c8zopkeSweOqsf3AfiW1yoPQOmxBdva8F7kKysV9bCJLsn yzO/1tEXfuzISyqm3vGlGzhgU64KgHHa3b+6yN5zQXRe2mmTXK6IjVBw1 nCj5ItzsomrfGA2zRdgELnUJPTeSYfFpR2RbtXdAfrDswJYxyCQgcdplp A=;
IronPort-PHdr: =?us-ascii?q?9a23=3AskYdZhZN0ujmSu44axSIDdn/LSx94ef9IxIV55?= =?us-ascii?q?w7irlHbqWk+dH4MVfC4el21QaXD5rS9+lJjazQvryzEWAD4JPUtncEfdQMUh?= =?us-ascii?q?IekswZkkQmB9LNEkz0KvPmLklYVMRPXVNo5Te3ZE5SHsuteFTOuXC0qzgfBk?= =?us-ascii?q?a3OQ98PO+gHInUgoy+3Pyz/JuGZQJOiXK9bLp+IQ/wox/Ws5wdgJBpLeA6zR?= =?us-ascii?q?6arw=3D=3D?=
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0DDCgDW4m1f/4oNJK1fHQEBPAEFBQE?= =?us-ascii?q?CAQkBFYFPAoFQUQeBSS8sCod4A417mHaBLoElA1ULAQEBDQEBLQIEAQGESwK?= =?us-ascii?q?CLgIkNAkOAgMBAQsBAQUBAQECAQYEbYVcDIVyAQEBAQIBEi4BATgECwIBCBU?= =?us-ascii?q?BMDIlAgQBEggTB4VQAw4gAaw2AoE5iGF0gTSDAQEBBYUyGIIQCYE4AYJxijw?= =?us-ascii?q?bgUE/gVSCGDU+hBQBEBgCg0iCLZALCDaLLoFlmWUKgmeaeaEQkwigBQIEAgQ?= =?us-ascii?q?FAg4BAQWBVDqBV3AVO4JpUBcCDZIQilZ0NwIGCgEBAwl8i3MBJ4ENATFfAQE?=
X-IronPort-AV: E=Sophos;i="5.77,302,1596499200"; d="scan'208";a="567964616"
Received: from alln-core-5.cisco.com ([173.36.13.138]) by alln-iport-2.cisco.com with ESMTP/TLS/DHE-RSA-SEED-SHA; 25 Sep 2020 12:31:10 +0000
Received: from XCH-ALN-002.cisco.com (xch-aln-002.cisco.com [173.36.7.12]) by alln-core-5.cisco.com (8.15.2/8.15.2) with ESMTPS id 08PCV9ER028524 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=FAIL); Fri, 25 Sep 2020 12:31:10 GMT
Received: from xhs-rcd-001.cisco.com (173.37.227.246) by XCH-ALN-002.cisco.com (173.36.7.12) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 25 Sep 2020 07:31:09 -0500
Received: from xhs-rtp-001.cisco.com (64.101.210.228) by xhs-rcd-001.cisco.com (173.37.227.246) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 25 Sep 2020 07:31:09 -0500
Received: from NAM10-DM6-obe.outbound.protection.outlook.com (64.101.32.56) by xhs-rtp-001.cisco.com (64.101.210.228) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Fri, 25 Sep 2020 08:31:09 -0400
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=B03Lx1WWgs/lAP4lP55yUJ+/fJSwv4jqQTwwNzoqSB6n4YXp9Ow5TI8vseIy8yOfQxopbaGLSD82qXkoFX0UVvI510peJDuTeVc57x8K+JpwNuduHZc+j+KfqGLkPZYlS5Jpv/ijrmCgCmZlE0LIwRUCe72+cZUcRTlJg+k0RVcv+Ds/EOk8m/BmZ1wb+wtcC8rUMDN/83P0h5/7FcuxZXmxYLaaRV6Efm1bE6fQ1hlJPtbHMjLdh36Xa6Hd3o5B71p4zTM7BE23xAV2gG2PwdtNa7qWvCNX14pDctgYV8YB11NyPsIZ50A3DQp/yuc0+AuhVEsZmE8S7LnMtxxN6Q==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pWIDI+8LmjNnS7ggurk7US524i1TP8B0lHl1Npe7X+U=; b=i5ucrvI4rxpz0ykiMWQi1jV9rncjzbSsWh8b//znELEZoaGgbk+yzkVy3eayKRci2ZBjbJwbMbQCTHCkChv8tP4REAj4mf1MSRF5C2h50sP1DH3fyU423LCyN4A8zSKJTeGf/glt2JbKwHJkybohE2cCP73UDwzOZY590RGyG6ryaxv4hHpSOfAaJA/dvh7iqtNPv9dOtZkQdU4bS+dxYoD/70DTipAD/JXiqE0jwAYLxBrQqRoziBbSDm8DTYpKtZt9DPz778xmBE5r0QirmxfMOAa4dGZPqHH9OrlKcsWh3nId8yAevWWNl7tLNmrMgljI04QBW7UFxlNWO/TOcA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cisco.com; dmarc=pass action=none header.from=cisco.com; dkim=pass header.d=cisco.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.onmicrosoft.com; s=selector2-cisco-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pWIDI+8LmjNnS7ggurk7US524i1TP8B0lHl1Npe7X+U=; b=fkZaK+LpfORM31ncXHoocdTozftgd2Y5wdij0sBz4EhbT6TvX87cHqB6GIKBZRvvqaYuSh3bc2UPlD/fMV/TnvGmbHYS/qcaFP+VppLCjUHpbc/aqc7EomGAjO57M6Jbp5/gkybrrE6dWhJwBl+14jkHOF8rPh7ScjizL06CIwQ=
Received: from MWHPR11MB2032.namprd11.prod.outlook.com (2603:10b6:300:2b::13) by MWHPR11MB1374.namprd11.prod.outlook.com (2603:10b6:300:24::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3412.22; Fri, 25 Sep 2020 12:31:08 +0000
Received: from MWHPR11MB2032.namprd11.prod.outlook.com ([fe80::9966:b539:a9f4:60dc]) by MWHPR11MB2032.namprd11.prod.outlook.com ([fe80::9966:b539:a9f4:60dc%9]) with mapi id 15.20.3412.025; Fri, 25 Sep 2020 12:31:08 +0000
From: "Per Andersson (perander)" <perander@cisco.com>
To: Kent Watsen <kent+ietf@watsen.net>, "netconf@ietf.org" <netconf@ietf.org>
Thread-Topic: [netconf] restconf collections
Thread-Index: AQHWkyuf0+ZngtdDJ0S+NlOVghodU6l5Mm9+
Date: Fri, 25 Sep 2020 12:31:07 +0000
Message-ID: <MWHPR11MB203298217DBFF10B168BC693DB360@MWHPR11MB2032.namprd11.prod.outlook.com>
References: <01000174c4eeb24e-c672ebe0-23a1-4f0e-afba-5b44d932e0ed-000000@email.amazonses.com>
In-Reply-To: <01000174c4eeb24e-c672ebe0-23a1-4f0e-afba-5b44d932e0ed-000000@email.amazonses.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: watsen.net; dkim=none (message not signed) header.d=none;watsen.net; dmarc=none action=none header.from=cisco.com;
x-originating-ip: [173.38.220.54]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: ba323a37-55b2-4981-6e15-08d8614ee0df
x-ms-traffictypediagnostic: MWHPR11MB1374:
x-microsoft-antispam-prvs: <MWHPR11MB1374A2034377E316E45C9BD9DB360@MWHPR11MB1374.namprd11.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: oZjtfWYXNinmSOILm90JYLhvGakyQ1hXe4jP3j99q/7vIlwfLpY/I8+ci5WfcwTCfHGQX/c9BWdl9XB8mS1zDjq3M3mX3xbCkx3GI0tARAWRHqEmE0lBu844975Xz4N1oHzXNhmwWYYuVTGXkMdDF8/r+BNoFDiEkFJ7+STgkb9WQsDi4RLnobIkZK2BCMUiyZTcHqQBO3/egZkHCS6mezLZkVwXU7VdsMH8xz5DXk+WlngCVIlfXK1BZS30w9pHxzh+Al0vxf6j/t4OB9gWkyYytAyciwRLCKD3oueibvs6VaOL/PcUFJq/zc/qF3oSxeLXtvuaFMFUdeZoYLGM/tGkRZR2CxE/19vYJjBZW1LrySY37TiIUeUlWLPSEBVE
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MWHPR11MB2032.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(136003)(396003)(346002)(39860400002)(376002)(9686003)(71200400001)(186003)(26005)(55016002)(8676002)(91956017)(76116006)(66946007)(2906002)(66476007)(66556008)(83380400001)(5660300002)(110136005)(33656002)(8936002)(86362001)(52536014)(316002)(478600001)(6506007)(66446008)(64756008)(7696005); DIR:OUT; SFP:1101;
x-ms-exchange-antispam-messagedata: 5468gq1cIjb9sPxmGNyxLBEqs6V4/jmXjc4vjo431d85ITlGGLoZrfGg9MWnVxm25T3f3vTpAW5Imuxj7Wz893XibHvudJed9Wwt19QR85uicXryH6/spbJeAWxpnJvL34w6ymcroneLiXV434oQIF10cZnqOYfZCe1ALhEw8NZ6ENdJ95Hq79j0prsOTBOw5F0xjXk8mCzhXXaG2sq1B7ps5G5+/+aUGhmFmbQlNkiVReg9bJT28HvJGezpRhr6AgLWJDwPFFVujzak8DE/RE1Rl6bHZXfOkt4FYxF35tzjwlEZ81DQrS0h3qKhl6IsmhUcokoEw5kO8qYG1MID6Um/q5UPQqfL3ejeY0wibxgMrOxdDoh+8tYs06bMuV53zloKItd6eud6BHbUIoS1BL4F8iz/kUudLI9l0jO23UIHuml6j1S0U4rQa1IS9d/nt6xkMduvCA2gfPftHmF0jruywp2qbdrsg4EE7HN5TWKoOvyxIZC3K9SxLUzPcbQHxXhrB60yffRxE2CJorDRy6gNQgBS6YU81SAj6/xj9TQkyHG/yiPx/DE0Q6uBwpSKJlk4wi7yZts72VFk3Powc0z0Mj+C28SkoOoQxUtOEFZQ99DIJqVinDmILkie/DdlPD2+RMSOPvl7WTjcwwasTg==
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MWHPR11MB2032.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ba323a37-55b2-4981-6e15-08d8614ee0df
X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Sep 2020 12:31:07.9248 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 5ae1af62-9505-4097-a69a-c1553ef7840e
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 8TpDnxbp508Y8elLQNxP/IFDTiULHGeFcyLBVocpuR/CP6Lyw8yNpT8zSXb3CTo6XJtSvu3+mAaTc9qSpeJTaQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR11MB1374
X-OriginatorOrg: cisco.com
X-Outbound-SMTP-Client: 173.36.7.12, xch-aln-002.cisco.com
X-Outbound-Node: alln-core-5.cisco.com
Archived-At: <https://mailarchive.ietf.org/arch/msg/netconf/KCeY98B7BAOajajc7Ii4B-_NrCY>
Subject: Re: [netconf] restconf collections
X-BeenThere: netconf@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: NETCONF WG list <netconf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/netconf>, <mailto:netconf-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/netconf/>
List-Post: <mailto:netconf@ietf.org>
List-Help: <mailto:netconf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/netconf>, <mailto:netconf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 25 Sep 2020 12:31:24 -0000

From: Kent Watsen <kent+ietf@watsen.net> on Sent: Friday, September 25, 2020 13:03:
> The restconf-collections draft [1] expired long ago, and yet the problem remains.
>
> What is the problem?  In my opinion, the problem is more narrow in scope than before.  Specifically, I feel that it is limited to just "config false" lists, which tend to contain an enormous number entries.  Classic examples include "audit logs" and "device logs".
>
> I do not feel that "config true" lists are a problem because 1) if it's possible for the client to configure a list, then it should be no problem for the client to handle getting it back and 2) clearly this is what everyone is doing today, unless using proprietary solutions, as RFC 8040 offers no option.
>
> It's understood that supporting "config true" lists could improve performance some, but this seems more like a "nice to have" compared to supporting "config false" lists, where returning all list entries in a single response risks exhausting internal memory buffers resulting in failures, and thus solving this problem seems more of a "must have".  Not to mention, handling "ordered-by user" lists would be very difficult.
>
> [As an aside, how are people handling large "config false" lists today?  Is each list-entry streamed out as a "notification"?  Isn't there still always a need to have at least the most-recent list entries, which could still be a lot, on the server?]
>
> The net-net is that I'm thinking we should brushoff the restconf-collections draft, and I'm wondering to what extent the WG agrees.

I agree it should be brushed off.

I have been looking at restconf-next on github where this has been mentioned.
Will it be part of restconf-next or a separate effort?

Today we implement limit and offset as proprietary extensions to RESTCONF.
It would be good to have them (or some other pagination mechanism)
standardized.

If I recall correctly, we have use cases for config true list sizes in the thousands
or even tens of thousands.


> Regarding solution, I suggest a two-fold approach:
>
>  1) only support query parameters (e.g., offset, limit, sort, where) if a "config false" list is queried directly.  E.g., GET {+restconf}/ds/ietf-datastores:operational/example-module:widgets/widget?offset=20?limit=50.  Note that RESTCONF does not allow the list itself to be a resource target is exactly why this makes sense.

I would suggest supporting (at least) pagination for any list.

What is the point of selective support? As a user I would find it strange.


>  2) any "config false" list that is a descendent of the resource target returns a placeholder response, rather than its contents.  Specifically, a top-level query on <operational> would NOT return any "config false" list entries.  As for the placeholder response, perhaps it could return metadata, something like: { "example-module:widgets": { "widget": [ { "@": { "ietf-collections:redacted": [None] } } ] } }.  To be backwards-compatible, it could be this list "entry" is only returned if the client sends a special query parameter (e.g., ?collections-supported) or, alternately, the server implicitly always only returns at most one list-entry (e.g., as if ?limit=1 was passed).

It feels like the implementation could be simpler with the implicit ?limit=1
behaviour. Even with pagination the list could be large and perhaps
streamed out which would make collection of metadata more challenging.

Another possibility is to let the client handle such queries, instead of
replacing list contents with a placeholder let the client set ?depth=1
and effectively omit list contents..


--
Per