Re: [icnrg] Some thoughts on architectural choices for Manifests

"David R. Oran" <> Thu, 06 August 2020 20:12 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id AECC83A0E8B for <>; Thu, 6 Aug 2020 13:12:22 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id b9R6ZKnt4pV4 for <>; Thu, 6 Aug 2020 13:12:20 -0700 (PDT)
Received: from ( [IPv6:2607:fca8:1530::c]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 47CBE3A0E85 for <>; Thu, 6 Aug 2020 13:12:20 -0700 (PDT)
Received: from [] ([IPv6:2601:184:407f:80ce:820:b927:bcd2:91ed]) (authenticated bits=0) by (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id 076KCDK4015088 (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256 verify=NO); Thu, 6 Aug 2020 13:12:15 -0700
From: "David R. Oran" <>
To:, "Marc Mosko" <>, "Christopher Wood" <>
Cc: ICNRG <>
Date: Thu, 06 Aug 2020 16:12:08 -0400
X-Mailer: MailMate (1.13.1r5701)
Message-ID: <>
In-Reply-To: <alpine.OSX.2.21.2008051903040.98272@uusi>
References: <> <alpine.OSX.2.21.2008051903040.98272@uusi>
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Archived-At: <>
Subject: Re: [icnrg] Some thoughts on architectural choices for Manifests
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Information-Centric Networking research group discussion list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 06 Aug 2020 20:12:23 -0000

I’m still digesting this. Consider my comments more as questions, 
although all of you who know me also are aware of my predilection to 
couch questions as objections.

I also admit to not really understanding it completely after a couple of 
read-throughs. Some of this may be due to having to purge my translation 
cache of my usual understanding of some of the terms you have chosen. 
Feel free to respond, ponder for why the explanation isn’t getting 
through well to me, or just ignore.

On 5 Aug 2020, at 22:33, wrote:

> Hi Dave and Marc,
> great to have your "unpacking"! After the ICNRG interim meeting I had
> started to write down my viewpoint, yesterday I revised it and I was
> about to revise it again - maybe I should just blast it out, same
> TL;DR disclaimer.
> Summary: I distinguish between
> - packet (data object)

I don’t like using the term “packet” this way for a networking 
audience, since this is not its usual meaning as a layer 3 PDU that is 
intended to be interpreted hop-by-hoop and not just end-to-end.

> - real blob (a packet's content)
> - virtual blob (content as defined by a manifest packet)
I guess I don’t understand what is “virtual” about virtual blobs. 
Usually “virtual” means/implies “not bound by the resource 
constraints of the underlying system”. Also, I would have thought what 
is in the manifest are *pointers*  not virtualized representations of 
the object. If the mental model you’re proposing is like the 
translation/indirection of virtual memory systems, I think that’s 
likely to be just as limiting/problematic as the iNode analogy we just 

> Manifests are the mechanism to define virtual blobs - full stop. The
> "full stop" excludes object composition.
Not sure what you mean by “object composition”. I thought what you 
were trying to exclude from FLIC’s design was the ability to handle 
sets/directories/groups/enumerations of independent objects.

> Relating to Dave's enumeration of options, my writeup would belong to 
> #2
> ("Keep the current FLIC design, but build in some extensibility
> features"). It would be a very lean FLIC, though.
> Regarding "direct embedding" - I also think that this is essential to
> have (I had called it inlining).
Great! Let’s take this as a hard requirement then. It would also be 
good to have this work recursively - allowing the embedding/inlining of 
Manifests as well as primitive objects in case we have a multi-level 
architecture where some semantics are in “top level”, while others 
just have pointer arrays.

> Regarding Marc's comments on annotation (thanks, too!): my writeup
> belongs to your group 3 ("Separate Data/Content objects within the
> FLIC world") in a strict way, the reason being that the encoding size
> of a manifest's annotations in general cannot be bound, hence would be
> the first candidate to benefit from virtual blobs as offered by the
> manifest construct.

This makes me more than a little nervous. My mental model of annotations 
is that they are not generalized metadata but only “application 
independent hints to drive the fetching process of Interest/Data 
exchanges”. I’m even more nervous about accommodating generalized 
metadata or “sidecar” data along the lines of resource forks used by 
some file systems in manifests. This has the risk of complicating the 
security aspects quite considerably, as it means that either keying used 
for application data may need separate contexts for the data versus the 
annotations, or conversely that you need separate keying for the base 
manifest than the annotations. To quote the NSA, “We never met a data 
we didn’t want to capture and examine”.

Coming back to what started me down this rabbit hole is that in my metal 
model annotations are not arbitrarily big and hence tight limitations on 
their size (e.g. no bigger than a single  SHA256 hash or a few of them) 
would not bother me in the least.

> If the encoding is small enough, the "separate
> data structure" can be "directly embedded". I guess from your text
> that such embedding is not equal to your "option 2" (parallel groups
> in one manifest), but it could be close as you wrote yourself.

> So here it goes, perhaps not a full manifest manifesto, but laying out
> an argument for keeping FLIC minimal and version-resistant.
> best, c
> ---
> First, I re-tell the FLIC story in new terms (no reference to inodes)
> for positioning FLIC and what it achieves. Then I will look into the
> "parallel datastructure" concern and how directory tree organization
> should be handled. I also include a grammar for such a lean FLIC.
> As a preparation I introduce "derivation functions" which take some
> packets (signed data objects) and produce new packets.

Are you sure you meant to say this? One purpose of Manifests is that you 
don’t want to sign individual packets, especially in hash-named 
schemes like CCNx. Perhaps this confusion is due to my mis-understanding 
of how you use the term “packet”. Don’t we agree that whether an 
input is already signed or not, the end result is that the individual 
pieces pointed to by the Manifest are not signed, but the signature over 
the Manifest covers all the pieces of the object. I *think* just signing 
the Manifest covers the needed security/trust requirements (assuming the 
Manifest is constructed by the same entity who contracted the original 
data) - Marc might weigh in here.

> The first
> derivation function of interest is "make_virtual_blob(h_1,..h_n)"
> whose result is exactly one packet of type manifest that contains the
> given sequence of packet hash values, these packet being either a
> normal data object ("real blob") or a manifest spec.
What about Merkle trees of Manifests?

> A packet with a manifest spec defines a "virtual blob" which is data
> of any size. The hash name of the manifest packet becomes the name of
> the virtual blob; the virtual blob inherits the signature of the
> manifest packet.
I’m not sure I understand what it means to “inherit” the 
signature. Do you mean “is incorporated into the overall signature 
calculation”? It seems you want to put a signature on each Manifest 
piece of a larger tree of manifest chunks, but I’m clearly missing 
something since that wouldn’t work.

>   real_blob_pkt.content() = justs tap into the named data object
>   virtual_blob_pkt.content() = gather all content by traversing the 
> manifest tree
>   note that for real blobs, real_blob_pkt.content() equals 
> real_blob_pkt.payload
>   (introducing the notation of field access vs a procedure)
> A manifest is a claim, namely that there exists a virtual blob which
> is the concatenating of all referenced content: it's a claim
> (attribution to a given manifest) that can only be verified by
> materializing the virtual blob. The claim made by a manifest can fail
> for several reasons, one being that some of the included hash values
> never had a corresponding data packet or that some of the referenced
> packets have been forgotten, or that the manifest spec is broken.
I’m not sure you can ever determine whether a given hash value has or 
ever had a corresponding data packet. You only know after the fact that 
you tried to fetch and it failed, but the networking uncertainty 
principle says that you can never tell the difference between 
nonexistent data and failure to fetch without connectivity to ask the 
original creator of the data. Also, depending on how Manifest indexing 
is done, it may be necessary to have bogus hashes in order to represent 
objects with “holes” in them.

> A second derivation function is "get_virtual_blob_content_size(h)".
> Again, it is a claim about a manifest that can only be verified by
> materializing the virtual blob. But unlike the first function, this
> function is not essential for defining the virtual blob, it is mere
> decoration (meta data). There are many more decorations:
> vblob_size_in_records
> vblob_size_in_UTF8_chars
> vblob_tree_depth
> vblob_number_of_leaf_blobs
> creation_time_in_sec

So, pulling this apart and using my proposed criteria for what kinds of 
annotations we want to permit in a Manifest, I would say:
> vblob_size_in_records -
no, since records are a purely application construct

> vblob_size_in_UTF8_chars
why is this different from size in bytes? And why would I care for the 
purposes of driving a fetch algorithm?

> vblob_tree_depth
yes, possibly useful

> vblob_number_of_leaf_blobs
yes, probably useful. If you have this and you require full packing of 
blobs, the depth can be inferred/calculated from this value.

> creation_time_in_sec
Isn’t this in the base protocol? And why seconds and not NTP time?

> The last example stands for a decoration that cannot be verified.
Define “verified”. You can certainly verify if (a)it’s in the 
future and hence impossible, or (b)it’s earlier than the creation time 
of the base object pieces (which may or may not be carried in the Data 
object portion messages of the fetch protocol).

> My suggestion is to leave decorations outside the FLIC draft, except
> for the possibility to have decorations. That is, it's ok to use a
> manifest for defining resource forks. I will come back to size
> limitations in just a moment.
I’m certainly open to this, as long as the extension machinery is 
clear, and the rules for how these are secured are specified up front so 
we don’t have an explosion of independent security contexts as people 
define extensions.

> Resource forks are a well known pattern that also map well to the
> networking world, for example layered video encodings,
So, I don’t want to argue this, other than pointing out that nobody I 
know treats layered video encodings as resource forks. One could 
generalize all the way to “tracks” in a ISOFF video file (which 
aren’t done as resource forks since they have a thing like a Directory 
style Manifest at the front to tell you what the tracks are and how to 
index them). Also, at least video and audio tracks tend to be 
interleaved to optimize disk I/O.

> labeling
> information for a video etc. These additional resources are optional
> and not needed to build the virtual blob abstraction - they enrich
> it. And these additional (parallel) resources all have the problem of
> potentially being desynchronized - after all they are all claims that
> the consuming end has to handle with caution.

Sure, yes. Again, I think of them as optimizations for transport, 
storage and possibly caching purposes that don’t change the semantics 
of how any of the data in the actual objects are interpreted by the 
receiving application.

> Size is one among many of such decorations. Yes, one could request
> that all manifest MUST include the size resource fork. But then the
> FLIC draft must explain how a virtual blob consumer should handle
> manifests that lack that resource fork, or what to do when traversal
> reveals that a size value was wrong. The answer will probably include
> a sentence like "if one of the correctness checks fails you must
> revert to deriving that property yourself", which is another way of
> saying that handling decoration data is decoration-specific.

again, if you treat these as “hints” that could be wrong or not 
present, the right expectations are set. Also, I would hope these would 
all be in the category of generalized fetch libraries and not specific 
to an application and whose values would not normally be exposed to 
application code.

> So better
> have a separate "FLIC decoration draft" explaining the various ways of
> adding decorations, including size: do it per-manifest (= store the
> sum of all referenced subtrees sizes), or as a single table at the
> root manifest that keeps the (size) labels for all nodes of the tree
> etc.

I am of course happy to leave the specification of the individual 
annotations/decorations to separate drafts as long as we agree on the 
exact extension mechanism and that each one of these not have a 
completely different parsing structure. In other words, if we decide 
they are attached to each pointer in the hash pointer array as TLVs, 
then each specification is only the semantics of what each T means, and 
what L’s are allowed. If we decide that they are parallel arrays 
corresponding to the pointer arrays, then we should require that the 
array traversal be the same for all of them, and hopefully in lock-step 
with the traversal and matching of the base pointer arrays. If we decide 
it’s a completely independent object (I hope not, but anyway if we 
do), we should define in the base spec what the allowed structure of 
such objects are.

> My other argument (beside diminishing decorations as claims) for not
> being concerned about potentially non-synchronous data is that code
> for producing and consuming manifests itself is constantly evolving
> and is not synchronized, due to bugs or partial implementations.
Agree. That’s why anything we do ought to support versioning.

> Even
> when resource forks themselves would be synchronized with a manifest's
> content, the softwares (plural) will not - the next version of a
> producer software may introduce wrong decorations expected by all
> existing consumer software. Said differently: Synchronization loops
> pass through software - making some decorations mandatory will not
> prevent desynchronization.
Agree, so I would definitely say up front that none of the 
decorations/annotations shall be mandatory to implement, and certainly 
not mandatory to use.

> Note that decorations can be useful for applications _and_ for the
> manifest traversal algorithms - I did not keep these two usages apart,
> there is only one decorations field. As a side effect, manifests thus
> provide a "decorated virtual blob" abstraction.
I’m arguing the exact opposite, that these only affect storage, 
transport, and possibly caching in an application agnostic way and that 
anything that is to be consumed by an application be handled as actual 
application data with its own Name and corresponding Manifest.

> When it comes to the question of using manifests to express
> directories or similar trees of components, we should not confound
> decoration with object composition.
Also agree. I see these as separate as well.

> It looks as if one could use the
> (hash) table as a table for sub-component data. One argument here is
> that we will bump into the size limitation of (manifest) packets
> wherefore we should compose objects out of virtual blobs _and_ the
> composition spec also be stored in a separate virtual blob. For
> example, what if the file name is larger than fits in one packet?
> It is much cleaner to store the mapping table (that maps from file
> attributes like names to virtual blobs) in a virtual blob.
While I agree in general with the size concern, the example is not a 
good one, since the name has to fit in an Interest packet when fetching 
the file, so it had better fit in the Manifest.

> Potential confusion can arise when we build a directory tree where
> files and directories are all stored in virtual blobs.
Yes, groups of groups, collections of collections, sets of sets, etc. 
are not the same thing as fragments of a single larger logical data 

> Isn't this
> a manifest tree of manifest trees? No, it is not because they operate
> at different levels. When traversing a manifest tree, we extract hash
> references from the manifest's fields.

> When traversing directories,
> we extract hash references from inside the virtual blob (defined by
> that manifest). These are very different memory zones.
> Here is a writeup in form of a grammar/definitions:
> packet := length-limited data object (NDN, CCN)
>          I assume that a data object has type information such that
>          "real blob" can be distinguished from "manifest packet". 
> Having
>          a hash name and retrieving the data object will tell us what
>          kind of packet it is. Anything else (e.g., tagging hash
>          pointers whether they are pointing to real blobs or a 
> manifest
>          pkt is a claim and thus unreliable. It's fine to keep such
>          tags as a decoration but these are not needed to implement
>          the virtual blob abstraction.
> r_pkt := packet with TLV for "real blob"
> m_pkt := packet with TLV for "manifest"
> r_pkt.payload = any                              # the "real blob"
> m_pkt.payload = manifest_spec
> manifest_spec := deco + cont_seq                 # two fields, must 
> fit in one packet
> deco          := None | hashval | inlined
> cont_seq      := sequence of (hashval | inlined)
> vblob_content(h) is defined as follows:
>           if type(fetch(h)) == r_pkt:
>                   fetch(h).payload
>           elif type(fetch(h)) == m_pkt:
>                   concat( vblob_content(j) for all j in 
> fetch(h).payload.cont_seq )
>                   # if j is inlined content, this is used instead of 
> vblob_content(j)
>           else:
>                   fail
> # The following would be part of a separate "FLIC decor draft":
> vblob_deco(h, fork_name) is defined as follows:
>           if type(fetch(h)) == r_pkt:
>                   None
>           elif type(fetch(h)) == m_pkt:
>                   d = fetch(h).payload.deco
>                   if d == None:
>                        None
>                   elif d is inlined:
>                        d
>                   else:
>                        vblob_content(d).access(fork_name)
>           # the mapping for fork_name to the resource, labeled as 
> "access" above,
>           # would also be part of that separate "FLIC decor draft"
> # How would composition e.g., directories look? We show content access
> # via a file's name and assume that the directory data structure maps
> # a file name to some hash value (which names a virtual blob
> # containing the file's content). This content access has no
> # similarities with above content retrievals, as it calls
> # vblob_content() twice:
> dir_get_content(dir_hash_val, file_name):
>         vblob_content( vblob_content(dir_hash_val).access(file_name) )
> # we can decorate files because they are stored as virtual blobs:
> dir_get_deco(dir_hash_val, file_name):
>         vblob_deco( vblob_content(dir_hash_val).access(file_name) )
> Composition does not require new NDN/CCN types: it's an application
> level issue whether it wants to define a "type decoration", or put a
> magic number (type) inside each directory virtual blob, and how it
> encodes the composition spec.
> Conclusions: The sole role of manifests should be to abstract away
> from blob size limitations, providing transparent access to virtual
> blobs (vblobs) of arbitrary size.

So, I don’t think that’s the only purpose of Manifests. I’ll post 
a separate message on that subject hopefully justifying my views.

> A manifest defined as a list of hash
> values, together with manifest recursion, are all the tricks we need -
> plus name constructors for forwarding reasons, not discussed in this
> writeup. "Decorations" (meta data) are important for speedup reasons,
> so there must be a hook for them. As decorations may require the
> support of vblobs themselves, the suggestion is that the hook comes in
> form of a single hash value (that references a vblob), instead of
> hardwiring decoration information into the manifest packet.

That may be ok, as long as certain restrictions are enforced, like
- requiring congruent traversal with the traversal of the pointers of 
the base Manifest structure,
- being application-independent so they are interpretable by things that 
don’t understand the application data
- that they not be of unbounded size, like the base application objects 
so they don’t become a dumping ground for random bits that should have 
been separately named and fetched.
- that they not require additional security contexts beyond those 
applied to secure the base manifest data.

> Inlining
> is still possible as an optimization. It's also possible to use the
> manifest construct for simply decorating a real or virtual blob.
If this recurses I sort of get off the bus. While it may be 
intellectually elegant to allow decorations to have decorations, we’d 
be stepping way over the bound that at least I had in my mind when 
proposing pointer annotations.

> For
> example, decorations could carry additional signatures, or type
> information outside NDN/CCN's TLV system. Composing objects (stored in
> vblobs) is different from glueing together blobs (to create
> vblobs). Similar to storing decoration data, composition should be
> layered on top of the vblob abstraction using one vblob for every
> component.
Ditto for this. We don’t want to migrate all of namespace and 
application design into the building and interpreting of Manifests.

Whew, that’s all I can think of for now.

I hope to post a separate message tomorrow on my view about how 
Manifests ought to be useful to elements other than the producer and 
consumer of the underlying data, such as Repos and forwarder caches.

> ---