Re: [netconf] Current activities on RESTCONF/NETCONF to support paging

I agree that this would be a useful problem to solve.

I also agree with Juergen that there will likely be consistency issues depending on implementation, particularly if the data is coming from <operational>.  But I suspect that this issue already exists for many non-trivial implementations (e.g. where the operational data is distributed to daemons, linecards, or remote slave devices).

Requiring that a consistent view of the conventional configuration datastores can be provided (without locking) might be a reasonable compromise.

Thanks,
Rob

-----Original Message-----
From: netconf <netconf-bounces@ietf.org> On Behalf Of Juergen Schoenwaelder
Sent: 14 March 2019 09:36
To: Wisotzky, Sven (Nokia - DE/Stuttgart) <sven.wisotzky@nokia.com>
Cc: netconf@ietf.org
Subject: Re: [netconf] Current activities on RESTCONF/NETCONF to support paging

One of the issues to consider is that depending on the datastore accessed, the data can change more or less rapidly and it is unclear whether you want servers to maintain potentially big amounts of state between requests. If you retrieve data in chunks, the question is whether you can simply put the data back together or the client side or whether this may lead to data that never really existed in that form on the server. With RESTCONF, etags may help to check whether the resource queried remains consistent (but then the server still has to be able to decide etag changes).

But backing up for this discussion of details, I agree that this is a problem worth solving by standardizing a common solution.

/js

On Thu, Mar 14, 2019 at 09:23:53AM +0000, Wisotzky, Sven (Nokia - DE/Stuttgart) wrote:
> All,
> 
> There was some discussion in earlier days (back in 2012/2014) to 
> support paging for extra-long lists. The main use-case is clearly to 
> improve support for interactive user-frontends, as loading entire 
> lists with potentially more than 100.000’s objects easily can become a 
> nightmare: loading time, memory consumption, crashes, …
> 
> For the sake of implementation, one would simply provide an attribute called “max-count”, “max-entries”, “max-elements” or “limit” – which basically defines the upper limit of list entries to be returned. Asking for limit=10 – even if the list contains a 100k the result would simply show 10. From an user-interface point of view, this might be enough – as users would see how the list entries look like to narrow down the search by adding additional search/filter criteria. So to protect the client system and improvements on loading time/memory consumptions – this might be just enough.
> 
> More advanced implementations might allow the NETCONF client to specify the “offset” which is basically the index of the first list entry to be returned. In such case, the client could continue to load further pages on customer demand (like dynamically while scrolling down the list or pressing on the next button). Even if people are not curious that much about memory consumption, even it should be possible to continue loading the remaining list in the background – while showing the first X items faster on the user-interfaces.
> 
> To make such paging solution even more complete, following use-cases are to be reviewed from a RESTCONF/NETCONF API point of view:
> /1/ How to figure out the number of list entries matching the given criteria for a given list, without the need to poll the entire list? This could be helpful to operators straight away, but also allows easier implementation for things like progress bars.
> /2/ Define a sorting criteria for search results /3/ Improve filtering 
> capabilities for subtree filters (e.g. contains operator)
> 
> I did not found any active DRAFT or RFC for this. That said, we can see various vendors implementing proprietary solutions for this in their RESTCONF stacks.
> Would really like to see an IETF defined approach for this, to avoid multi-vendor clients need to deal various vendors implementations.
> 
> /wiso

> _______________________________________________
> netconf mailing list
> netconf@ietf.org
> https://www.ietf.org/mailman/listinfo/netconf

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

_______________________________________________
netconf mailing list
netconf@ietf.org
https://www.ietf.org/mailman/listinfo/netconf