Re: [icnrg] Maximizing the utility of Manifests

[Marc Mosko <[e-mail]>]

From: "David R. Oran" <[e-mail]>
Date: Friday, August 7, 2020 at 7:29 AM
To: Marc Mosko <[e-mail]>, Christian Tschudin <[e-mail]>, Christopher Wood <[e-mail]>
Cc: ICNRG <[e-mail]>
Subject: Maximizing the utility of Manifests

I’m starting a new thread on this, since it’s coming at some of the architectural choices from a broader point of view than what the producers/consumers of application data may need in a Manifest. The implications, if you buy into my general proposition, touch on a number of aspects of Manifest design, including extensibility tradeoffs, what goes into the base required functionality, and how we view both security and privacy issues.

Here’s the basic premise. I believe we want Manifests to be useful to any element that stores, fetches, or caches the content described by a manifest. This includes both:

·         Repositories (in NDN terms “Repos”) that provide non-volatile storage of content produced elsewhere and act both as:
o    archival storage that can survive the temporary or possibly permanent unavailability of the original producer of data
o    A form of distributed cache in which content can be replicated and advertised into routing the same way the origin producer would.

·         Forwarder Caches (Content Stores) that can provide various optimizations beyond simple on-demand opportunistic caching to improve latency and/or robustness to downstream consumers. This would include watching for requested Manifests and in addition to caching them, using information in the Manifest to pre-fetch content likely to be requested by one or more downstream consumers.

I think I would say it like “A manifest data structure should enable a reader to correctly fetch the original data in the proper order with no external knowledge, assuming it as the proper security context for the manifest.  The security context of the manifest may be different than the security context of the data.”   A reader might not be able to reassemble the data, if it does not have that security context.  But if it can read the manifest, it can read, in-order, the data Data objects.

Both of these require that Manifests have, I believe at a minimum, the following characteristics:

1.    Manifests have Name Constructors so that these non-application elements are able to form Interests to fetch the content described in the Manifest without knowing application-specific namespace conventions.

2.    We have some generally understood annotations (or whatever we decide to call this restricted form of metadata) to give hints about:
o    optimal/desirable fetch order
o    relative importance for availability when producer is either not reachable or down
o    interdependence of pieces if partial and not total to help drive cache replacement policies
I do not believe that annotations should be able to change the correct traversal order to read the data in-order (which is currently the pre-order DFS traversal order).  As long as a reader can ignore annotations and still correctly re-assemble the original file, the annotations can say anything they want.

3.    Clear and simple binding to the “right” security context so that there is not a complex explosion of signing and encryption keys. My recommendation would to allow only two security contexts for Manifests
o    signed/encrypted by keys used for application data. Such data would not be available to the intermediaries descried above.
o    signed/encrypted under by the keys used for the Manifest itself. Such data would be available to any/all elements that are given the keys associated with the names in the Manifest’s namespace.
I view this as the manifest has a single security context, for the manifest.  The data can do whatever the data wants to do.  It would most likely be a separate security context.   A reader with the manifest security context knows the correct pointers for the data and can fetch the data, but the security context for reading the data is unrelated to the manifest.
Exception to the rule: maybe there is an annotation that tells the reader what the data context will be so it can start working on that before it receives any data.  But that is purely an optimization, it does not affect manifest processing at all.
Maybe we need to expand this a little:

  1.  The outer Manifest security context, usually on only a top-level manifest, is a public key signature.  This binds the manifest to a publisher and is used for trust purposes.  It also binds the ensuing hash tree to the signature, so subsequent manifest objects do not need a public key signature.  This signature may be the same as the data.  This security context is specified by the normal CCNx/NDN mechanisms for signing data.  There can be variations on this if one allows in-lining, as I mentioned in a prior email.
  2.  The manifest-specific security context.  This is the encryption context, though as we use AEAD mechanisms, it also provides some authentication too.  This could be as simple as a pre-shared AES key, or a group encryption mechanism, or a broadcast encryption mechanism, etc.
  3.  The Data security context.  Normally, under a Manifest system, the data Data objects do not need to be signed.
  4.  The data encryption context.  Encryption is done in whatever mechanism the publisher wants.  Personally, I think we should specify a standard way of doing this, or come up with a conversion of CMS, or adopt a standard record format.  Something.

4.    any other goop in a Manifest (e.g. general application metadata) is treated as if were any other transparently cached application data, although I argue elsewhere against trying to support general application metadata in Manifests.

I think having annotations that could point to side-car data (generally their own FLIC) is the way to go.  If an application wants to have, say, bibliographic data for something, have an annotation that says to look there.  I think these would generally only be used in the top-level manifest.   Or the application could always bake it in to the application data.

So, as a top level question, do people think these are important and/or reasonable requirements to accommodate in our Manifest architecture?

I’ll note that it is of course possible to design a completely separate data structure and named elements for use by Repositories and forwarders, but that seems to me to be way more complex than just viewing manifests in this broader context.

DaveO