Re: [Cellar] Matroska Elements to support frame side data

[Steve Lhomme <[e-mail]>]

I would go with the Track way as well. Primarily because storing a
string (which pretty much never changes) in each Block is a huge
waste.

Add extra data per Block is already supported using BlockAdditions.
There's already BlockAddID which correspond to Moritz' BlockMetadataID
and BlockAdditional which correspond to BlockMetadataString /
BlockMetadataBinary / BlockMetadataUInteger / BlockMetadataSInteger /
BlockMetadataFloat. It's like a codec where the CodecID defines how
the data in the binary blob should be interpreted.

The current system states that the additions are left to
interpretation to the codec. It was originally designed to hold the
lossless complement to lossy versions of Musepack. So in that case
it's really meant to be passed to the codec. I think we can expand
this system with keeping this default behaviour by default (albeit not
used anywhere) and have different ones on demand.
There's also an AlphaMode that also uses BlockAdditions to store the
alpha track. Which pretty much no info on how to do it....

As noted timecode may be a separate track (as originally intended) if
it is not related to the video frames (ie the timestamps doesn't
match).

This would look like this:
- Musepack lossless complement:
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 1
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 1
(same as BlockAddID) (default)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName:
"complement" (default)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType: 0
(Codec Complement data) (default)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 1
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
lossless part interpreted by the codec

- Alpha layer:
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 2
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 2
(same as BlockAddID)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "alpha"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType: 1
(Alpha layer data)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 2
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
alpha mask to apply on the video track

- RawCooked DPX data
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 3
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 3
(same as BlockAddID)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "rawcooked"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType:
0x1234567 (rawcooked identifier)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 3
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
DPX data defined by RawCooked

- Timecode storing
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 3
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 3
(same as BlockAddID)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "timecode"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType:
0x890ABCD (SMPTE TC identifier, can be another ID for different kind
of timecode)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 3
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
Timecode storage

That means the alpha mode would not be backward compatible with
existing files, because it requires non default values. But I don't
think anyone ever used this improperly defined feature.

The value of MaxBlockAdditionID is kept low on purpose. BlockAddID 1
was always for codec complement and thus 2 for the AlphaMode. But we
don't need to go much higher that than now that we have a mapping. If
there are Timecode AND Rawcooked it would be like this:
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 4
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 3
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "rawcooked"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType:
0x1234567 (rawcooked identifier)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 4
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "timecode"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType:
0x890ABCD (SMPTE TC identifier, can be another ID for different kind
of timecode)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 3
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
DPX data defined by RawCooked
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 4
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
Timecode storage

Le lun. 5 nov. 2018 à 10:29, Moritz Bunkus
<moritz=[e-mail]> a écrit :
>
> Hey,
>
> > In other thoughts on this suggestion, I think it could make it difficult
> > to easily understand if a file has a particular type of side data. For
> > instance if only a few Clusters somewhere in the Segment contain a
> > certain type of side data, it would require parsing every Cluster to know
> > what types of side data are available. This uncertainly wouldn’t be the
> > same issue if the side data was itself a Track.
>
> It's not entirely necessary to use a full track for side data. We can simply
> signal the presence of side data in the track headers and refer to it from
> the side data in the block groups. This would also mean we only have to
> store the string identifying the side data type once (in the track headers)
> instead of in each block.
>
> (I'll use "BlockMetadata" as the basis for all element names
> here. Initially I proposed "FrameMetadata", but "BlockMetadata" is fine
> with me, too.)
>
> For example:
>
> Tracks
> +- TrackEntry
>  +- TrackBlockMetadata (Master)
>   +- TrackBlockMetadataType (String, required)
>   +- TrackBlockMetadataID (Unsigned Integer, required)
>
> …
>
> Cluster
> +- BlockGroup
>  +- BlockMetadata (Master)
>   +- BlockMetadataID (Unsigned Integer, required, refers to existing
>      TrackBlockMetadataID in track headers)
>   +- BlockMetadataString (Unicode String, optional)
>   +- BlockMetadataBinary (Binary, optional)
>   +- BlockMetadataUInteger (Unsigned Integer, optional)
>   +- BlockMetadataSInteger (Signed Integer, optional)
>   +- BlockMetadataFloat (Float, optional)
>
> with the restriction that exactly one of (BlockMetadataString,
> BlockMetadataBinary, BlockMetadataUInteger, BlockMetadataSInteger,
> BlockMetadataFloat) must exist.
>
> Advantages as I see them:
>
> • Less overhead (no repeated string parsing required)
> • Quicker parsing (no repeated string parsing required)
> • Presence of meta data is known upfront
> • Not using a full-blown track for meta data would alleviate the need to
>   specify how all those track and block features (e.g. BlockDuration,
>   TrackDefaultDuration…) apply to a "meta data track".
>
> Kind regards,
> mosu
>
> _______________________________________________
> Cellar mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/cellar



-- 
Steve Lhomme
Matroska association Chairman