[netmod] JSON encoding of anydata: approximating an operational leaf-list
Jan Kundrát <jan.kundrat@cesnet.cz> Mon, 21 February 2022 10:44 UTC
Return-Path: <jan.kundrat@cesnet.cz>
X-Original-To: netmod@ietfa.amsl.com
Delivered-To: netmod@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AC78D3A0E6D for <netmod@ietfa.amsl.com>; Mon, 21 Feb 2022 02:44:17 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.698
X-Spam-Level:
X-Spam-Status: No, score=-0.698 tagged_above=-999 required=5 tests=[BAYES_05=-0.5, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MIME_QP_LONG_LINE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=cesnet.cz
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VNJFPk1oopXn for <netmod@ietfa.amsl.com>; Mon, 21 Feb 2022 02:44:11 -0800 (PST)
Received: from office2.cesnet.cz (office2.cesnet.cz [195.113.144.244]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 623823A0E74 for <netmod@ietf.org>; Mon, 21 Feb 2022 02:44:10 -0800 (PST)
Received: from localhost (ip-78-45-154-116.net.upcbroadband.cz [78.45.154.116]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by office2.cesnet.cz (Postfix) with ESMTPSA id CDCF540006A for <netmod@ietf.org>; Mon, 21 Feb 2022 11:44:07 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cesnet.cz; s=office2-2020; t=1645440247; bh=ei65DuWbkyisYAfh8j1k+l9aqiVMouWSs2JJu4d8/8o=; h=From:To:Subject:Date; b=dE2jSq/Y33irmpMlEaRQb/ezrOXDmKeR6as5/plHhmEe3CH/BZTsM7DhbULTSpy5u jk2ef9okG0y8j+fhJ8brw04qwjvWfYSTvDkL4ySGa5WF3xxZNlKEKizR5H0y82arYe 2HhASs18EJTfku8as5mLM7j8/RGU96EweyyIDxIcbJr7CFXniRABDvghOiaIsJfC/W JX+wopU0rkdnSdI3o6grE8NQ1WrNp3pSIqbQsfJsCunniZfwYNhC0m+oH8NlcOQzD6 eCatkiif5U36dJLYeJXjLXlXFqmcBzCk30/utQlbaheO9Ux9doiiwy7YYW2KF9fnGa 3qNkmfXfLlmXQ==
From: Jan Kundrát <jan.kundrat@cesnet.cz>
To: netmod@ietf.org
Date: Mon, 21 Feb 2022 11:44:07 +0100
MIME-Version: 1.0
Message-ID: <13b2c889-89f6-437b-b149-090bb9d144b7@cesnet.cz>
Organization: CESNET
User-Agent: Trojita/unstable-2020-07-06; Qt/5.15.3; xcb; Linux;
Content-Type: text/plain; charset="utf-8"; format="flowed"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://mailarchive.ietf.org/arch/msg/netmod/GMjIvI2AlXtpjj8P5yaorNIB42s>
Subject: [netmod] JSON encoding of anydata: approximating an operational leaf-list
X-BeenThere: netmod@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: NETMOD WG list <netmod.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/netmod>, <mailto:netmod-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/netmod/>
List-Post: <mailto:netmod@ietf.org>
List-Help: <mailto:netmod-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/netmod>, <mailto:netmod-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 21 Feb 2022 10:44:18 -0000
Hi, last year we published some work [1] about using IETF YANG-push for telemetry streaming in the context of optical networks. One of the use cases was continualy sending spectral scans, and for us that meant updating a list of roughly 15-20k items at least once per second. Think of this as a list of (frequency, power) pairs where the "frequency" can be implied, so the data was really a "a long vector of numbers". We were not able to use either `list` or a `leaf-list` directly for performance reasons. The root cause is that `leaf-list` traditionally used to be "an ordered set". It was changed to allow duplicate values in YANG 1.1, and that for ops data only. Still, various implementations try to look at the operational leaf-lists and helpfully compute diffs, they look for item moves, etc. This is not cheap for 15-20k items. In the end we used a model like this: leaf lowest-frequency { type frequency-ghz; config false; mandatory true; description "Lowest frequency slice in the `p` list."; } leaf step { type frequency-ghz; config false; mandatory true; description "Each subsequent `p` result is this far from the previous one."; } anydata p { config false; description "Measured power. Sorted by frequency, starting at `lowest-frequency`, step size `step`. Individual items are decimal64 numbers."; } The JSON-encoded data looked like this: { "something: { "lowest-frequency": 123456, "step": 1, "p": ["0.003", "0.001", "0.001", "0.24", "0.37"] } } Our internal implementation uses libyang, and we're now updating form libyang v1 to libyang v2, which assumes that our `p` has to be encoded as JSON object. This of course produces an invalid JSON: "p": {["0.003", "0.001", "0.001", "0.24", "0.37"]} I have a few questions: - Can `anydata` represent a JSON array directly? On one hand the standard says that "An anydata instance is encoded in the same way as a container, i.e., as a name/object pair", but then it also allows the `empty` thing via `[null]`. Or is it perhaps allowed only for *members* of this faux container? - In case `anydata` is not a good match, I suppose `anyxml` will work fine because the example in the RFC is already using an array. Perhaps we could also use JSON numbers directly without encoding as a string as well. - Is there a better approach than this one? Can I reasonably expect "all the YANG implementations out there" to stop looking at operational leaf-lists as essentially ordered sets? I think the answer is "no", but I am looking for suggestions. With kind regards, Jan [1] https://ieeexplore.ieee.org/document/9457112
- [netmod] JSON encoding of anydata: approximating … Jan Kundrát
- Re: [netmod] JSON encoding of anydata: approximat… Carsten Bormann
- Re: [netmod] JSON encoding of anydata: approximat… Carsten Bormann
- Re: [netmod] JSON encoding of anydata: approximat… Martin Björklund