Re: [netconf] restconf collections

Kent Watsen <> Mon, 28 September 2020 14:21 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id E1AB93A0538 for <>; Mon, 28 Sep 2020 07:21:50 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id O6sDr8YI5EwI for <>; Mon, 28 Sep 2020 07:21:49 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 08E953A08BB for <>; Mon, 28 Sep 2020 07:21:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=224i4yxa5dv7c2xz3womw6peuasteono;; t=1601302907; h=From:Content-Type:Mime-Version:Subject:Date:References:To:In-Reply-To:Message-Id:Feedback-ID; bh=A4F06zdf2bsYd8MFPbgk4FwL4JJmExK4pEDTNzorQ4s=; b=cHwZ8bsuB4oXEfVUepoLO+U3bvBrsVquBWSAZ51D9mpSUPqo56xqncvzJKyEAzk9 QsEncBxBh23sgYjEwB/ZAayHz98OrCjjR7aKnWLqvDdD9cvi4gdu6P7X0CQzNzNkOTD /fs/9R3UBEwFalrEjKrgyRlXlIe6hU5z2cSJZ7YA=
From: Kent Watsen <>
Content-Type: multipart/alternative; boundary="Apple-Mail=_71F18AF1-AEDA-429D-829B-85E1CB61A38C"
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.\))
Date: Mon, 28 Sep 2020 14:21:47 +0000
References: <> <> <> <> <>
To: "" <>
In-Reply-To: <>
Message-ID: <>
X-Mailer: Apple Mail (2.3608.
X-SES-Outgoing: 2020.09.28-
Archived-At: <>
Subject: Re: [netconf] restconf collections
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: NETCONF WG list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 28 Sep 2020 14:21:51 -0000

It seems that we need two I-Ds:

1) “restconf-lists"

	a) “update” RFC 8040 (and possibly RFC 8527)
	b) modify Section 3.5 (Data Resource) to state that “list” and “leaf-list” themselves (not just their entries) are “data resources”
	c) define impact on various HTTP methods GET, POST, PUT, DELETE, PATCH, etc.
	d) additionally define query parameters for the GET:
		i) limit: number to return.  Could name ‘count’ to match <get-bulk>
		ii) offset: where to begin.  Need to discuss if <get-bilk>’s “keys” is better
		iii) where: some xpath expression
		iv) direction: ascending | descending

2) “netconf-lists"

	a) “update” RFC 6241 (and possibly RFC 8526)
	b) introduce a <get-bulk> or <get-block> like RPC for paging through the entries of lists and leaf-lists.

Anyone interested in working on said I-Ds, please email me privately.

Kent. // as a contributor

> On Sep 27, 2020, at 11:12 AM, Andy Bierman <> wrote:
> On Sat, Sep 26, 2020 at 11:57 AM Kent Watsen < <>> wrote:
>> The <get-bulk> operation is not stateful.
>> <>
> This RPC is effectively 1-1 with RC query params approach being discussed.  I like it.
> The “count” param is like “limit”, and “list-test” is like a “where”.
> One difference is that <get-bulk> uses "last-keys” instead of an integral “offset”.   Any thoughts about the pros and cons?  I imagine performance depends on the backend database used.  For instance, SQL supports “offset” so, to support "last-keys” instead would entail a first query to determine the offset for a key.
> It would be unlikely that a distributed server (e.g. data can come from linecards)
> that it is efficient to find the Nth entry.
> Using a count approach is obviously vulnerable to additions and deletions to the dataset.
> If the count keeps changing then the relative Nth entry also keeps changing.
> This causes duplicates or missed entries in the paginated data.  Using key values
> instead of a count this source of inaccuracy is eliminated.  Both approaches
> are vulnerable to "moved" entries (e.g. user-ordered list entry changes position).
> This can happen in theory but (1) user-ordered config=false data is very rare,
> (2) the "move" operation for user-ordered data is very rarely used and (3)
> an entry would have to be moved just at the time the walk was in progress.
> Missing is a “sort”.  That said, of the four params mentioned (limit, offset, where, sort), it might be least important.  At least, a general sort is…and then there is the nasty issue of sorting a user-sorted list.  A likely must-have fallback would be “direction” so, e.g., queries return the most-recent (not the oldest) entires in a time-series list (e.g., a log).  A “direction” parameter would also work well on a user-sorted list.
> sort is much more expensive to implement correctly and makes more
> sense when a stateful snapshot approach is used. Sorting each chunk
> vs. the entire data-set is not that useful.
>> There are 3 completely different approaches:
>>   1) stateless
>>   2) stateful
>>   3) stateful snapshot
>> How to efficiently iterate through data that is changing constantly?
>> This problem is harder than it looks.
> Agreed.  I recommend we NOT try to solve that problem.
> But we are solving that problem to some degree.
> Operational state is changing constantly and iterating through a list
> vs. a single GET usually makes the problem worse.
> K.
> Andy