Re: [netconf] Current activities on RESTCONF/NETCONF to support paging

[Andy Bierman <[e-mail]>]

On Thu, Mar 14, 2019 at 3:24 AM Rob Wilton (rwilton) <[e-mail]>
wrote:

> I agree that this would be a useful problem to solve.
>
> I also agree with Juergen that there will likely be consistency issues
> depending on implementation, particularly if the data is coming from
> <operational>.  But I suspect that this issue already exists for many
> non-trivial implementations (e.g. where the operational data is distributed
> to daemons, linecards, or remote slave devices).
>
> Requiring that a consistent view of the conventional configuration
> datastores can be provided (without locking) might be a reasonable
> compromise.
>
>
This thread has already mentioned some of the reasons the RESTCONF
Collections draft got shelved.

 1) this is not a RESTCONF-specific problem, so a RESTCONF-specific
solution will not solve the problem
 2) there is no agreement on how much state a server should maintain during
the "get-bulk walk".
 3) some features like sorting require all data to be retrieved within the
server and sorted, instead of streaming the data
 4) position-based "get-next" is error-prone in dynamic operational state;
it requires expensive locking in configuration

I agree a standard solution would be beneficial to client developers, if
all 4 issues could be solved


Thanks,
> Rob
>
>
Andy



>
> -----Original Message-----
> From: netconf <[e-mail]> On Behalf Of Juergen
> Schoenwaelder
> Sent: 14 March 2019 09:36
> To: Wisotzky, Sven (Nokia - DE/Stuttgart) <[e-mail]>
> Cc: [e-mail]
> Subject: Re: [netconf] Current activities on RESTCONF/NETCONF to support
> paging
>
> One of the issues to consider is that depending on the datastore accessed,
> the data can change more or less rapidly and it is unclear whether you want
> servers to maintain potentially big amounts of state between requests. If
> you retrieve data in chunks, the question is whether you can simply put the
> data back together or the client side or whether this may lead to data that
> never really existed in that form on the server. With RESTCONF, etags may
> help to check whether the resource queried remains consistent (but then the
> server still has to be able to decide etag changes).
>
> But backing up for this discussion of details, I agree that this is a
> problem worth solving by standardizing a common solution.
>
> /js
>
> On Thu, Mar 14, 2019 at 09:23:53AM +0000, Wisotzky, Sven (Nokia -
> DE/Stuttgart) wrote:
> > All,
> >
> > There was some discussion in earlier days (back in 2012/2014) to
> > support paging for extra-long lists. The main use-case is clearly to
> > improve support for interactive user-frontends, as loading entire
> > lists with potentially more than 100.000’s objects easily can become a
> > nightmare: loading time, memory consumption, crashes, …
> >
> > For the sake of implementation, one would simply provide an attribute
> called “max-count”, “max-entries”, “max-elements” or “limit” – which
> basically defines the upper limit of list entries to be returned. Asking
> for limit=10 – even if the list contains a 100k the result would simply
> show 10. From an user-interface point of view, this might be enough – as
> users would see how the list entries look like to narrow down the search by
> adding additional search/filter criteria. So to protect the client system
> and improvements on loading time/memory consumptions – this might be just
> enough.
> >
> > More advanced implementations might allow the NETCONF client to specify
> the “offset” which is basically the index of the first list entry to be
> returned. In such case, the client could continue to load further pages on
> customer demand (like dynamically while scrolling down the list or pressing
> on the next button). Even if people are not curious that much about memory
> consumption, even it should be possible to continue loading the remaining
> list in the background – while showing the first X items faster on the
> user-interfaces.
> >
> > To make such paging solution even more complete, following use-cases are
> to be reviewed from a RESTCONF/NETCONF API point of view:
> > /1/ How to figure out the number of list entries matching the given
> criteria for a given list, without the need to poll the entire list? This
> could be helpful to operators straight away, but also allows easier
> implementation for things like progress bars.
> > /2/ Define a sorting criteria for search results /3/ Improve filtering
> > capabilities for subtree filters (e.g. contains operator)
> >
> > I did not found any active DRAFT or RFC for this. That said, we can see
> various vendors implementing proprietary solutions for this in their
> RESTCONF stacks.
> > Would really like to see an IETF defined approach for this, to avoid
> multi-vendor clients need to deal various vendors implementations.
> >
> > /wiso
>
> > _______________________________________________
> > netconf mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/netconf
>
>
> --
> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> Phone: [phone]         Campus Ring 1 | 28759 Bremen | Germany
> Fax:   [phone]         <https://www.jacobs-university.de/>
>
> _______________________________________________
> netconf mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/netconf
> _______________________________________________
> netconf mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/netconf
>