Re: [netconf] restconf collections

Andy Bierman <> Sun, 27 September 2020 15:12 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 1C3AC3A0FB6 for <>; Sun, 27 Sep 2020 08:12:22 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.887
X-Spam-Status: No, score=-1.887 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, T_SPF_PERMERROR=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id aydGN7YX8Qhj for <>; Sun, 27 Sep 2020 08:12:20 -0700 (PDT)
Received: from ( [IPv6:2a00:1450:4864:20::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 1BA873A0FB5 for <>; Sun, 27 Sep 2020 08:12:19 -0700 (PDT)
Received: by with SMTP id s205so6250772lja.7 for <>; Sun, 27 Sep 2020 08:12:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=TBiV/8W3GAtjTgK1fZ42tPkdxr7vald6tFYEdEYc3os=; b=m3dHBjni1ZD4Zj1ZH6YOpAaC9zKkql5lH3eolOKAMVP1bmBhcA+JXnEQxN2COUbFfg /EQl0qekEJFakU4dk3roV3v/FpvC8caLrhIOsYLUFen5fmHQqxdNHwdY/9FCIFzxfnA6 NCVVu9mcrl+BvLI8HMHOy57tdMREjA27cmzwzZb+4DqIVtJ+WRXeFTqQPdsWBKwJGlKr b0qfwMWeLbAlt3Mam52ECdZ+8eUtloOBfhsHwJDhg9BIVJtdchDkS1uvtzHIL8FtQlFr zw6S8bhgWqkSgeXkBTsdvdDZ/QILAYTOGdv3Fz4hHolEPkURYYcvGHzD1KGMd8Imjvn9 pMZw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=TBiV/8W3GAtjTgK1fZ42tPkdxr7vald6tFYEdEYc3os=; b=q9Aq4TPn4qlDfRPrqTs4QRqIlN0G1ygLCnpZXzVtLDzIstp6lbNJt/JWiDzx3VmUKf 9Q+72VNw0ZdnAGR98ZL72xageEtwEmqxefJ/xx2Yx/mCegPCUz9tArDoEgneVG/xkcoN icNhCWmwqHhpVYunUTRwcNb1FxeJoIq8cqq3M052G7d3J8HM/2+6UHof2hdE8zEGhyTH 1+HUEO8vytTbskO66Wg39IksJmwfUBJZJdbNDD+OwnpNDP6Xd2/JNfrTCWBbmBNg1vpj T0XVBuNGRlNNhCBk2EYZmRKqDEqS6GO0F8ppbFUDY5PkvZbF95iQ0YRJj2uG2QQqEq5G 1qWQ==
X-Gm-Message-State: AOAM532mPh6JSYxIIAyfCuWRRMfHhxyhTOmad5CiUzDL0/gSU/CASEpo KsJ4gM1LroTz+6EvRndtrNV8dA3EeJWUH4r/aYe9MiyC9yfhoQ==
X-Google-Smtp-Source: ABdhPJzhhQFEsz3O4a/xhfv8oyoUBJGj6oVG407VNyh+1ujcOmixJnBGDQMCM/La8llmu4S7HyH/2gUa/wg7cf0i32k=
X-Received: by 2002:a2e:a550:: with SMTP id e16mr4049657ljn.125.1601219537456; Sun, 27 Sep 2020 08:12:17 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <> <>
In-Reply-To: <>
From: Andy Bierman <>
Date: Sun, 27 Sep 2020 08:12:06 -0700
Message-ID: <>
To: Kent Watsen <>
Cc: "" <>
Content-Type: multipart/alternative; boundary="000000000000bcf3fd05b04cf951"
Archived-At: <>
Subject: Re: [netconf] restconf collections
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: NETCONF WG list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Sun, 27 Sep 2020 15:12:22 -0000

On Sat, Sep 26, 2020 at 11:57 AM Kent Watsen <> wrote:

> The <get-bulk> operation is not stateful.
> This RPC is effectively 1-1 with RC query params approach being
> discussed.  I like it.
> The “count” param is like “limit”, and “list-test” is like a “where”.
> One difference is that <get-bulk> uses "last-keys” instead of an integral
> “offset”.   Any thoughts about the pros and cons?  I imagine performance
> depends on the backend database used.  For instance, SQL supports “offset”
> so, to support "last-keys” instead would entail a first query to determine
> the offset for a key.

It would be unlikely that a distributed server (e.g. data can come from
that it is efficient to find the Nth entry.

Using a count approach is obviously vulnerable to additions and deletions
to the dataset.
If the count keeps changing then the relative Nth entry also keeps changing.
This causes duplicates or missed entries in the paginated data.  Using key
instead of a count this source of inaccuracy is eliminated.  Both approaches
are vulnerable to "moved" entries (e.g. user-ordered list entry changes
This can happen in theory but (1) user-ordered config=false data is very
(2) the "move" operation for user-ordered data is very rarely used and (3)
an entry would have to be moved just at the time the walk was in progress.

> Missing is a “sort”.  That said, of the four params mentioned (limit,
> offset, where, sort), it might be least important.  At least, a general
> sort is…and then there is the nasty issue of sorting a user-sorted list.  A
> likely must-have fallback would be “direction” so, e.g., queries return the
> most-recent (not the oldest) entires in a time-series list (e.g., a log).
> A “direction” parameter would also work well on a user-sorted list.

sort is much more expensive to implement correctly and makes more
sense when a stateful snapshot approach is used. Sorting each chunk
vs. the entire data-set is not that useful.

> There are 3 completely different approaches:
>   1) stateless
>   2) stateful
>   3) stateful snapshot
> How to efficiently iterate through data that is changing constantly?
> This problem is harder than it looks.
> Agreed.  I recommend we NOT try to solve that problem.

But we are solving that problem to some degree.
Operational state is changing constantly and iterating through a list
vs. a single GET usually makes the problem worse.

> K.