Re: [icnrg] Some thoughts on architectural choices for Manifests

[Marc Mosko <[e-mail]>]

On 8/7/20, 6:56 AM, "icnrg on behalf of David R. Oran" <[e-mail] on behalf of [e-mail]> wrote:

    On 6 Aug 2020, at 16:37, Marc Mosko wrote:

    > > We choose to make those descriptions use the same data structure. We 
    > > also choose that
    > > when reading a manifest, a reader does not know which pointers belong 
    > > to which sub-tree until it actually fetches a pointer (unless there 
    > > are annotations).
    > I think this is right, however isn’t it a good idea to enforce a 
    > canonical ordering so that you don’t need application knowledge to 
    > re-assemble the pieces of an object in the right order (or are you only 
    > referring to non-leaf manifests pieces)?

Yes, the manifest has an implicit traversal order to read the Data pointers in proper order.

    > > The manifest sub-tree is really just a
    > > FLIC representation of the manifest file and the data sub-tree (it's 
    > > not so much a tree as the in-order traversal of leaf pointers) is a 
    > > representation of the data file.
    > So, looking at my above comment in the context of this sentence we are 
    > saying the manifest ensures that there is a canonical ordering of leaf 
    > manifests, and in leaf manifests, the order of the pointers is congruent 
    > with the order of the pointed-to data bytes, right?

Any manifest may have both manifest pointers and data pointers, not just leaf manifests.  As it is now, there is only one class of pointers, and they may point to a Manifest or Data.  There is only one traversal order, and this order ensures that both the manifest tree and data pointers are visited in the proper order such that the data pointers are read in order.

The only way a leaf manifest is different than an interior manifest is that it only has data pointers.  But there is no easy way for a consumer to know it is a leaf manifest (maybe it could guess based on subtree sizes).

    > >
    > > In some FLIC arrangements, this can be made explicit.  For example, my 
    > > root manifest has two hash groups.  One group pointing to the manifest 
    > > tree and the second points to the data leaves.  Or, as is more common, 
    > > they are just mixed I one hash group.
    > >
    > Seems fine. Might there be advantages (either in flexibility or 
    > simplicity) in separating them?

The only reason to separate them is to have different Name Creators for the manifest subtree versus the data pointers.

The options are things like:

Mixed hashgroup: [MDMDDDMMDDMMDDD]
Separate: [MMMMMM] [DDDDDDDDD]
Interleaved: [DDD] [MMMM] [DDDDDD] [MMM]

Note that the M pointers are not the same between these options, as the traversal order is always pre-order DFS.  This means if you want to read data at the current node first, you want to put Data pointers before Manifest pointers.

    > > This would allow the NameCreator
    > > extension to specify different rules for different HashGroup types, or 
    > > a private producer could have their own secret rules for different 
    > > HashGroup types.  This allows an app to use different NameCreator 
    > > approaches for the manifest file and the data file.   It's like being 
    > > able to use typed name components so an app can have its own rules.

    > There is a certain danger in this if we expect to have manifests widely 
    > used by “third parties” other than the producer and consumer of a 
    > particular app. I also would like to stress the comment I made earlier 
    > that permitting large random amounts of application “metadata” to 
    > find their way into Manifests can be deleterious for both security and 
    > rational namespace design. I am fairly wedded to the notion that the 
    > only thing that belongs in Manifests are things that allow elements that 
    > fetch, store, or cache content to correctly fetch large objects (and in 
    > my view also be able to iterate over collections), and to do so without 
    > any private knowledge of application semantics.

I agree, I'd prefer to not have secret application data determine how to decode a manifest.  Annotations could give hints to optimized ways of reading things (e.g. a video player wants to skip ahead), but it should never change the canonical "if you traverse it like normal you get the original file" paradigm.  I was throwing out a few options to evaluate, not all of them are good.

Marc