Re: [netconf] restconf collections

[Kent Watsen <[e-mail]>]

It seems that we need two I-Ds:

1) “restconf-lists"

	a) “update” RFC 8040 (and possibly RFC 8527)
	b) modify Section 3.5 (Data Resource) to state that “list” and “leaf-list” themselves (not just their entries) are “data resources”
	c) define impact on various HTTP methods GET, POST, PUT, DELETE, PATCH, etc.
	d) additionally define query parameters for the GET:
		i) limit: number to return.  Could name ‘count’ to match <get-bulk>
		ii) offset: where to begin.  Need to discuss if <get-bilk>’s “keys” is better
		iii) where: some xpath expression
		iv) direction: ascending | descending

2) “netconf-lists"

	a) “update” RFC 6241 (and possibly RFC 8526)
	b) introduce a <get-bulk> or <get-block> like RPC for paging through the entries of lists and leaf-lists.


Anyone interested in working on said I-Ds, please email me privately.

Thanks,
Kent. // as a contributor


> On Sep 27, 2020, at 11:12 AM, Andy Bierman <[e-mail]> wrote:
> 
> 
> 
> On Sat, Sep 26, 2020 at 11:57 AM Kent Watsen <[e-mail] <mailto:[e-mail]>> wrote:
> 
>> The <get-bulk> operation is not stateful.
>>  http://www.netconfcentral.org/modulereport/yumaworks-getbulk <http://www.netconfcentral.org/modulereport/yumaworks-getbulk>
> This RPC is effectively 1-1 with RC query params approach being discussed.  I like it.
> 
> The “count” param is like “limit”, and “list-test” is like a “where”.
> 
> One difference is that <get-bulk> uses "last-keys” instead of an integral “offset”.   Any thoughts about the pros and cons?  I imagine performance depends on the backend database used.  For instance, SQL supports “offset” so, to support "last-keys” instead would entail a first query to determine the offset for a key.
> 
> 
> 
> It would be unlikely that a distributed server (e.g. data can come from linecards)
> that it is efficient to find the Nth entry.
> 
> Using a count approach is obviously vulnerable to additions and deletions to the dataset.
> If the count keeps changing then the relative Nth entry also keeps changing.
> This causes duplicates or missed entries in the paginated data.  Using key values
> instead of a count this source of inaccuracy is eliminated.  Both approaches
> are vulnerable to "moved" entries (e.g. user-ordered list entry changes position).
> This can happen in theory but (1) user-ordered config=false data is very rare,
> (2) the "move" operation for user-ordered data is very rarely used and (3)
> an entry would have to be moved just at the time the walk was in progress.
> 
>  
> Missing is a “sort”.  That said, of the four params mentioned (limit, offset, where, sort), it might be least important.  At least, a general sort is…and then there is the nasty issue of sorting a user-sorted list.  A likely must-have fallback would be “direction” so, e.g., queries return the most-recent (not the oldest) entires in a time-series list (e.g., a log).  A “direction” parameter would also work well on a user-sorted list.
> 
> 
> sort is much more expensive to implement correctly and makes more
> sense when a stateful snapshot approach is used. Sorting each chunk
> vs. the entire data-set is not that useful.
> 
>  
> 
> 
>> There are 3 completely different approaches:
>> 
>>   1) stateless
>>   2) stateful
>>   3) stateful snapshot
>> 
>> How to efficiently iterate through data that is changing constantly?
>> This problem is harder than it looks.
> 
> Agreed.  I recommend we NOT try to solve that problem.
> 
> 
> 
> But we are solving that problem to some degree.
> Operational state is changing constantly and iterating through a list
> vs. a single GET usually makes the problem worse.
> 
> 
> 
> K.
> 
> 
> Andy