Re: [netconf] restconf collections

Kent Watsen <kent+ietf@watsen.net> Fri, 02 October 2020 13:07 UTC

Return-Path: <01000174e96cd06f-4ecb06ac-c7ac-4233-bd5d-97d3dbc74830-000000@amazonses.watsen.net>
X-Original-To: netconf@ietfa.amsl.com
Delivered-To: netconf@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id ED4593A0FD5 for <netconf@ietfa.amsl.com>; Fri, 2 Oct 2020 06:07:39 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level:
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=amazonses.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nRsLxOOuIAfA for <netconf@ietfa.amsl.com>; Fri, 2 Oct 2020 06:07:38 -0700 (PDT)
Received: from a48-90.smtp-out.amazonses.com (a48-90.smtp-out.amazonses.com [54.240.48.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2A0243A0FCE for <netconf@ietf.org>; Fri, 2 Oct 2020 06:07:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=224i4yxa5dv7c2xz3womw6peuasteono; d=amazonses.com; t=1601644056; h=From:Message-Id:Content-Type:Mime-Version:Subject:Date:In-Reply-To:Cc:To:References:Feedback-ID; bh=Tjej7FQkTXaybFIRW7DFowIvxdpq8Y9BxBinR0Dklds=; b=To0WI60NnJBHBFIqhKcRJ1noxCKVCeR0G4GZLcHh90/UYRzfw8ulcIBEjZQEf5Zz RJhFMZW5AhwUSxCJ0dt48D49Xnk1SNu8V55nTL/3NoAEGeYS9JT3laUsFGjix+7KaBK YufJ/ZeEp5lYGnMOp5sfJK5colKLC4LP3K2lO2Pc=
From: Kent Watsen <kent+ietf@watsen.net>
Message-ID: <01000174e96cd06f-4ecb06ac-c7ac-4233-bd5d-97d3dbc74830-000000@email.amazonses.com>
Content-Type: multipart/alternative; boundary="Apple-Mail=_54C7951C-B7F7-4214-9B6B-195E94A21AD2"
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Date: Fri, 02 Oct 2020 13:07:36 +0000
In-Reply-To: <20201002.082331.2242297379846191383.id@4668.se>
Cc: Olof Hagsand <olof@hagsand.se>, "netconf@ietf.org" <netconf@ietf.org>
To: Martin Björklund <mbj+ietf@4668.se>
References: <01000174e4610ae6-d1488376-ae7e-4d95-ba7e-005fc44b9592-000000@email.amazonses.com> <20201001.165548.1275939966226069939.id@4668.se> <01000174e5ba8cc5-39edb446-48d9-4896-a6b5-dcf80f962762-000000@email.amazonses.com> <20201002.082331.2242297379846191383.id@4668.se>
X-Mailer: Apple Mail (2.3608.80.23.2.2)
X-SES-Outgoing: 2020.10.02-54.240.48.90
Feedback-ID: 1.us-east-1.DKmIRZFhhsBhtmFMNikgwZUWVrODEw9qVcPhqJEI2DA=:AmazonSES
Archived-At: <https://mailarchive.ietf.org/arch/msg/netconf/BtRJLt0BhAlkp-d41AwZnGs2FEU>
Subject: Re: [netconf] restconf collections
X-BeenThere: netconf@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: NETCONF WG list <netconf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/netconf>, <mailto:netconf-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/netconf/>
List-Post: <mailto:netconf@ietf.org>
List-Help: <mailto:netconf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/netconf>, <mailto:netconf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 02 Oct 2020 13:07:40 -0000

>> 3) Here we’re talking about querying a potentially large repository of
>> logs that are at rest, and full table scans should avoided.  I suggest
>> that we’re dealing with something new here.
> 
> Perhaps we don't understand each other.  Here's an pseudo-code example
> of what I envision:
> 
>  GET /interfaces/interface
>  filter  'oper-status = "up" and statistics/in-errors > 0'
> 
> This is XPath 1.0, applied to one instance at the time.

Of course I understand Xpath filters, other relevant examples here:

	- https://tools.ietf.org/html/rfc5277#section-5.2 <https://tools.ietf.org/html/rfc5277#section-5.2>
	- https://tools.ietf.org/html/rfc8040#appendix-B.3.6 <https://tools.ietf.org/html/rfc8040#appendix-B.3.6>

The point was/is that supporting full-Xpath for stream-filters is easy, or rather doesn’t entail special consideration, as already each event is in memory for an Xpath analysis.  Thus, pointing to existing usage as evidence that unconstrained Xpath can be used everywhere, including for potentially very-large CF-lists, is not a given, as it would be performance-limiting to bring every list-entry into application-level memory in order to use an Xpath library.  In this case, it is very much desired to push all the querying logic to the DB-backend.


>> But when it comes to CF data, doing a full table scan is impractical
>> and index-backed queries can make the difference between the client
>> waiting seconds vs minutes.  This is why I was initially trying to
>> focus on just solving the CF problem, where the pain is acute.
> 
> Well, it is less likely that an implementation uses a feature-rich (in
> terms of query language) DB to store all its state, than its config.
> So it would probably be easier to implement this using "an index" for
> config than for config false...

Really?  ...my experience is seems opposite.  

A number of server implementations I’m aware of persist the entire configuration as a flat-file (e.g., pretty-print XML or JSON).  They bring it into memory and act on it using a feature-rich library (e.g., Yangson).  In my view, this approach hits the low-bar in terms of what constitutes a “database”. 

On the flip side, handling large CF-lists (e.g., logs) is where servers pretty much have to use a real database.  An RDB (relational DB) is common, but a TSDB (time-series DB) <https://en.wikipedia.org/wiki/Time_series_database> is better suited.

Perhaps we’re saying the same thing (yes, config APIs tends to have better query expressions)…it might just be your last line regarding “index” that’s throwing me (as clearly the DBs storing large CF-lists are having indexes).


>> That said, I propose we do “both”.  <snip/>
> 
> I agree.

Okay, we’ll shoot for a two-pronged approach.



Separately, for the RC-list draft only, the plan is to declare “list” and “leaf-list” nodes (not their entries) as data resource targets.  We only need to do this for GET, but I assume that it's better to cover all the HTTP operations - thoughts?


K.