Re: [netconf] Current activities on RESTCONF/NETCONF to support paging

["Wisotzky, Sven (Nokia - DE/Stuttgart)" <[e-mail]>]

Hi Jürgen,

I am not sure if this is so much different from retrieving huge lists using get or get-config today. Capturing information, XML rendering and transmission will take time - so nobody would be surprised, that receiving a list with +100k entries easily takes several minutes. In such situation, it really depends on the server implementation - if concurrent changes impact the result of get/get-config operation on-the-fly, making the result inconsistent.

Clearly a NETCONF/RESTCONF server could avoid such problems by taking a short-term log while capturing and caching all the information, while the XML rendering and transmission would use that snapshot which would be fully consistent. But to implement a NC/RC server that way, comes at some expense of memory footprint which is typically quite limited on network equipment and is likely one of the reasons why NETCONF XPATH support is rarely supported in the industry.

That being said, the only way for a multi-vendor NETCONF client to ensure data consistency is to lock the running datastore while executing a get-config operation.

Based on this, the same could easily be applied to paging. So if paging is simply used to make the user-interface showing tables faster while loading the remaining table in the background, the netconf server could lock the datastore for the entire operation to enforce a better level of consistency.

However, as per my initial post below, just defining a limit (aka max-count) is a very efficient initial step to improve loading time and avoid crashes. And the user-interface could be implemented in a way to first just ask for 100 entries to have something to show, while after this asking the entire list in the background. Just not a big fan of this, because if would cause some overhead.

About the number of list entries, clearly one could put this number for any list into the state model. But again, I believe it is much better to have a generic approach for this.

/wiso

On 14.03.19, 10:35, "Juergen Schoenwaelder" <[e-mail]> wrote:

    One of the issues to consider is that depending on the datastore
    accessed, the data can change more or less rapidly and it is unclear
    whether you want servers to maintain potentially big amounts of state
    between requests. If you retrieve data in chunks, the question is
    whether you can simply put the data back together or the client side
    or whether this may lead to data that never really existed in that
    form on the server. With RESTCONF, etags may help to check whether the
    resource queried remains consistent (but then the server still has to
    be able to decide etag changes).
    
    But backing up for this discussion of details, I agree that this is a
    problem worth solving by standardizing a common solution.
    
    /js
    
    On Thu, Mar 14, 2019 at 09:23:53AM +0000, Wisotzky, Sven (Nokia - DE/Stuttgart) wrote:
    > All,
    > 
    > There was some discussion in earlier days (back in 2012/2014) to support paging for extra-long lists. The main use-case is clearly to improve support for interactive user-frontends, as loading entire lists with potentially more than 100.000’s objects easily can become a nightmare: loading time, memory consumption, crashes, …
    > 
    > For the sake of implementation, one would simply provide an attribute called “max-count”, “max-entries”, “max-elements” or “limit” – which basically defines the upper limit of list entries to be returned. Asking for limit=10 – even if the list contains a 100k the result would simply show 10. From an user-interface point of view, this might be enough – as users would see how the list entries look like to narrow down the search by adding additional search/filter criteria. So to protect the client system and improvements on loading time/memory consumptions – this might be just enough.
    > 
    > More advanced implementations might allow the NETCONF client to specify the “offset” which is basically the index of the first list entry to be returned. In such case, the client could continue to load further pages on customer demand (like dynamically while scrolling down the list or pressing on the next button). Even if people are not curious that much about memory consumption, even it should be possible to continue loading the remaining list in the background – while showing the first X items faster on the user-interfaces.
    > 
    > To make such paging solution even more complete, following use-cases are to be reviewed from a RESTCONF/NETCONF API point of view:
    > /1/ How to figure out the number of list entries matching the given criteria for a given list, without the need to poll the entire list? This could be helpful to operators straight away, but also allows easier implementation for things like progress bars.
    > /2/ Define a sorting criteria for search results
    > /3/ Improve filtering capabilities for subtree filters (e.g. contains operator)
    > 
    > I did not found any active DRAFT or RFC for this. That said, we can see various vendors implementing proprietary solutions for this in their RESTCONF stacks.
    > Would really like to see an IETF defined approach for this, to avoid multi-vendor clients need to deal various vendors implementations.
    > 
    > /wiso
    
    > _______________________________________________
    > netconf mailing list
    > [e-mail]
    > https://www.ietf.org/mailman/listinfo/netconf
    
    
    -- 
    Juergen Schoenwaelder           Jacobs University Bremen gGmbH
    Phone: [phone]         Campus Ring 1 | 28759 Bremen | Germany
    Fax:   [phone]         <https://www.jacobs-university.de/>