[icnrg] Maximizing the utility of Manifests

"David R. Oran" <daveoran@orandom.net> Fri, 07 August 2020 14:29 UTC

Return-Path: <daveoran@orandom.net>
X-Original-To: icnrg@ietfa.amsl.com
Delivered-To: icnrg@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id 9376D3A0D07 for <icnrg@ietfa.amsl.com>; Fri, 7 Aug 2020 07:29:39 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id 770OZvq0ueWw for <icnrg@ietfa.amsl.com>; Fri, 7 Aug 2020 07:29:37 -0700 (PDT)
Received: from spark.crystalorb.net (spark.crystalorb.net [IPv6:2607:fca8:1530::c]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4105C3A0847 for <icnrg@irtf.org>; Fri, 7 Aug 2020 07:29:37 -0700 (PDT)
Received: from [] ([IPv6:2601:184:407f:80ce:820:b927:bcd2:91ed]) (authenticated bits=0) by spark.crystalorb.net (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id 077ETUar017185 (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256 verify=NO); Fri, 7 Aug 2020 07:29:32 -0700
From: "David R. Oran" <daveoran@orandom.net>
To: "Marc Mosko" <mmosko@parc.com>, "Christian Tschudin" <christian.tschudin@unibas.ch>, "Christopher Wood" <caw@heapingbits.net>
Cc: ICNRG <icnrg@irtf.org>
Date: Fri, 07 Aug 2020 10:29:25 -0400
X-Mailer: MailMate (1.13.1r5701)
Message-ID: <E728B5F3-9CC0-4B29-884A-A81B408BC1AD@orandom.net>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="=_MailMate_8DC78008-9516-47F5-B8EE-7CA16216CB8B_="
Content-Transfer-Encoding: 8bit
Archived-At: <https://mailarchive.ietf.org/arch/msg/icnrg/5Sqamud4Fv91ld5hqI50Y5DyWtY>
Subject: [icnrg] Maximizing the utility of Manifests
X-BeenThere: icnrg@irtf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Information-Centric Networking research group discussion list <icnrg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/icnrg>, <mailto:icnrg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/icnrg/>
List-Post: <mailto:icnrg@irtf.org>
List-Help: <mailto:icnrg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/icnrg>, <mailto:icnrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Fri, 07 Aug 2020 14:29:40 -0000

I’m starting a new thread on this, since it’s coming at some of the 
architectural choices from a broader point of view than what the 
producers/consumers of application data may need in a Manifest. The 
implications, if you buy into my general proposition, touch on a number 
of aspects of Manifest design, including extensibility tradeoffs, what 
goes into the base required functionality, and how we view both security 
and privacy issues.

Here’s the basic premise. I believe we want Manifests to be useful to 
any element that stores, fetches, or caches the content described by a 
manifest. This includes both:

- Repositories (in NDN terms “Repos”) that provide non-volatile 
storage of content produced elsewhere and act both as:
	- archival storage that can survive the temporary or possibly permanent 
unavailability of the original producer of data
	- A form of distributed cache in which content can be replicated and 
advertised into routing the same way the origin producer would.

- Forwarder Caches (Content Stores) that can provide various 
optimizations beyond simple on-demand opportunistic caching to improve 
latency and/or robustness to downstream consumers. This would include 
watching for requested Manifests and in addition to caching them, using 
information in the Manifest to pre-fetch content likely to be requested 
by one or more downstream consumers.

Both of these require that Manifests have, I believe at a minimum, the 
following characteristics:

1. Manifests have Name Constructors so that these non-application 
elements are able to form Interests to fetch the content described in 
the Manifest without knowing application-specific namespace conventions.

2. We have some generally understood annotations (or whatever we decide 
to call this restricted form of metadata) to give hints about:
	- optimal/desirable fetch order
	- relative importance for availability when producer is either not 
reachable or down
	- interdependence of pieces if partial and not total to help drive 
cache replacement policies

3. Clear and simple binding to the “right” security context so that 
there is not a complex explosion of signing and encryption keys. My 
recommendation would to allow only two security contexts for Manifests
	- signed/encrypted by keys used for application data. Such data would 
*not* be available to the intermediaries descried above.
	- signed/encrypted under by the keys used for the Manifest itself. Such 
data would be available to any/all elements that are given the keys 
associated with the names in the Manifest’s namespace.

4. any other goop in a Manifest (e.g. general application metadata) is 
treated as if were any other transparently cached application data, 
although I argue elsewhere against trying to support general application 
metadata in Manifests.

So, as a top level question, do people think these are important and/or 
reasonable requirements to accommodate in our Manifest architecture?

I’ll note that it is of course possible to design a completely 
separate data structure and named elements for use by Repositories and 
forwarders, but that seems to me to be way more complex than just 
viewing manifests in this broader context.