Re: [Cellar] matroska and side data vs timecode

[Steve Lhomme <[e-mail]>]

Le mar. 29 oct. 2019 à 17:04, Dave Rice <[e-mail]> a écrit :
> I drafted a pull request at https://github.com/cellar-wg/matroska-specification/pull/348 that builds upon the new side data structure defined above. This pull request registers a BlockAddIDType for a 64 bit binary expression of timecode as defined by SMPTE ST12-1:2014 (which is mostly hours, minutes, seconds, frames, a drop frame flag, and other a few other flags and binary groups). This would allow timecode to be associated with Matroska frames such as:
>
> <Segment>
>  <!-- skipped elements here -->
>  <Tracks>
>   <!-- lots of other track data here -->
>   <TrackEntry>
>    <BlockAdditionMapping>
>     <BlockAddIDValue>2</BlockAddIDValue>
>     <BlockAddIDName>SMPTE ST 12-1 timecode</BlockAddIDName>
>     <BlockAddIDType>121</BlockAddIDType>
>    </BlockAdditionMapping>
>  </Tracks>
>  <!-- then within the Cluster which stores audiovisual data within frames in Blocks, there is -->
>  <Cluster>
>   <BlockGroup>
>    <Block> { a video frame } </Block>
>    <BlockAdditions>
>     <BlockMore>
>      <BlockAddID>2</BlockAddID>
>      <BlockAdditional> { SMPTE ST12-1:2014 64-bit binary representation of a 07:32:54;18 } </BlockAdditional>
>     </BlockMore>
>    </BlockAdditions>
>   </BlockGroup>
>   <BlockGroup>
>    <Block> { the next video frame } </Block>
>    <BlockAdditions>
>     <BlockMore>
>      <BlockAddID>2</BlockAddID>
>      <BlockAdditional> { SMPTE ST12-1:2014 64-bit binary representation of a 07:32:54;19 } </BlockAdditional>
>     </BlockMore>
>    </BlockAdditions>
>   </BlockGroup>
>  </Cluster>
> </Segment>
>
> With the BlockAdditions, BlockMore, BlockAddID, and BlockAdditional Elements, I think this would add 18 bytes per timecode expression. Adding timecode to one hour of PAL video (25 frames * 3600 seconds/hour) would add about 1.5 MB to the file.

That seems like a big overhead. Is the 64 bits value necessary ? It
seems 32 bits might be sufficient (saving 4 bytes). Maybe 40 if you
need extra flags.

> Implementation Considerations:
>
> Thinking ahead of this work, I think it would be worthwhile to add a dedicated section to the Matroska specification on timecode and describe two methods for adding a reference timecode to a Matroska Segment.
>
> 1. Using tags. Make an official tag “TIMECODE” to store a string of the timecode expression that correlates to the Matroska timestamp of 0. This approach is limited to storing timecode that is incremental and continuous. Defaults for the incrementation and the behavior of the timecode could be described as part of the tag definitions and feasibly another tag or tags could be reserved for timecode flags when the timecode behavior is not a default. Defining a use of a “TIMECODE” tag would provide the benefit of standardizing an existing practice in FFmpeg’s Matroska muxer, where rewrapping from a timecode-supportive format to Matroska carries the timecode value over as a tag.

In general I'm not in favor of using tags for data that may be needed
when remuxing, especially tied to a timestamp (yes, I mean timestamp).
If the first frame is damaged, does the timecode apply to the second
frame ? If the first frame timestamp wasn't 0, how do you know the
difference you have to apply ?

If you want something that will survive this scenario you have to tie
this value to the first timestamp. When remuxing, if timestamps are
altered of portions at the beginning of the file are removed, you need
to edit the first timecode. It may be better to store in the TrackInfo
with both the first timestamp and timecode values. It may not work
when stripping the audio (as Tobias pointed out). So it may even be
put in the SegmentInfo instead of the TrackInfo.

You may also set the value in Chapters (which are not tied to a
track). There's already a start timestamp. You could add the timecode
for this timestamp. That works for NLE sources as well, each
non-linear part would be a chapter with its timestamp translation. A
good NLE system would also tag this chapter with all kinds of data
about the source. (Matroska was also designed for this case in mind).

> 2. Using BlockAdditions as described above. This adds more overhead as each timecode value would be written, but would support non-continuous timecode. In FFmpeg, timecode is sometimes carried as side-data such as from the decklink device via https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0946c0ec177dc48ef0677f890aa42d95e667c417. I’d be interested if the Block Addition Mapping described above could be used to carry the timecode side data from the decklink device (or other incoming stream with timecode side data) and store the result within the written Matroska BlockGroups.

That's the no-brainer option. It just works. It's just less optimal
that one value (or a few for NLE sources).

> Comments welcome.
>
> Thanks much,
> Dave Rice
> _______________________________________________
> Cellar mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/cellar



-- 
Steve Lhomme
Matroska association Chairman