[Cellar] matroska and side data vs timecode

[Dave Rice <[e-mail]>]

Hi cellar working group,

I wanted to share some recent and ongoing developments within https://github.com/cellar-wg/matroska-specification <https://github.com/cellar-wg/matroska-specification>.

Some background of recent Matroska side data work:

Recently a pull request at https://github.com/cellar-wg/matroska-specification/pull/287 was merged into the Matroska specification which adds a structure to identify, store, and describe side data to Matroska frames. This adds a block mapping to the TrackEntry that identifies the side data to a registry entry such as:

<Segment>
 <!-- skipped elements here -->
 <Tracks>
  <!-- lots of other track data here -->
  <TrackEntry>
   <BlockAdditionMapping>
    <BlockAddIDValue> {arbitrary value used to link BlockAddID (see the Cluster below) } </BlockAddIDValue>
    <BlockAddIDName>  {a human readable name for that side data}.   </BlockAddIDName>
    <BlockAddIDType>  {a registered ID for that type of side data}  </BlockAddIDType>
   </BlockAdditionMapping>
 </Tracks>
 <!-- then within the Cluster which stores audiovisual data within frames in Blocks, there is -->
 <Cluster>
  <BlockGroup>
   <Block> { some data, probably an audio or video frame } </Block>
   <BlockAdditions>
    <BlockMore>
     <BlockAddID> {arbitrary value used to link BlockAddIDValue (see above) } </BlockAddID>
     <BlockAdditional> { insert side data here, if the BlockAddID value is >=2 then look up what the data is in the BlockAdditionMapping } </BlockAdditional>
    </BlockMore>
   </BlockAdditions>
  </BlockGroup>
 </Cluster>
</Segment>

Within BlockMore, if the BlockAddID = 1 then the Codec Mapping defines what is within BlockAdditional as side data, but if the BlockAddID is >=2, then the identity of the BlockAdditional side data is defined within the BlockAdditionalMapping of the associated TrackEntry.

Active work on Matroska and timecode:

I drafted a pull request at https://github.com/cellar-wg/matroska-specification/pull/348 that builds upon the new side data structure defined above. This pull request registers a BlockAddIDType for a 64 bit binary expression of timecode as defined by SMPTE ST12-1:2014 (which is mostly hours, minutes, seconds, frames, a drop frame flag, and other a few other flags and binary groups). This would allow timecode to be associated with Matroska frames such as:

<Segment>
 <!-- skipped elements here --> 
 <Tracks>
  <!-- lots of other track data here -->
  <TrackEntry>
   <BlockAdditionMapping>
    <BlockAddIDValue>2</BlockAddIDValue>
    <BlockAddIDName>SMPTE ST 12-1 timecode</BlockAddIDName>
    <BlockAddIDType>121</BlockAddIDType>
   </BlockAdditionMapping>
 </Tracks>
 <!-- then within the Cluster which stores audiovisual data within frames in Blocks, there is -->
 <Cluster>
  <BlockGroup>
   <Block> { a video frame } </Block>
   <BlockAdditions>
    <BlockMore>
     <BlockAddID>2</BlockAddID>
     <BlockAdditional> { SMPTE ST12-1:2014 64-bit binary representation of a 07:32:54;18 } </BlockAdditional>
    </BlockMore>
   </BlockAdditions>
  </BlockGroup>
  <BlockGroup>
   <Block> { the next video frame } </Block>
   <BlockAdditions>
    <BlockMore>
     <BlockAddID>2</BlockAddID>
     <BlockAdditional> { SMPTE ST12-1:2014 64-bit binary representation of a 07:32:54;19 } </BlockAdditional>
    </BlockMore>
   </BlockAdditions>
  </BlockGroup>
 </Cluster>
</Segment>

With the BlockAdditions, BlockMore, BlockAddID, and BlockAdditional Elements, I think this would add 18 bytes per timecode expression. Adding timecode to one hour of PAL video (25 frames * 3600 seconds/hour) would add about 1.5 MB to the file.

Implementation Considerations:

Thinking ahead of this work, I think it would be worthwhile to add a dedicated section to the Matroska specification on timecode and describe two methods for adding a reference timecode to a Matroska Segment.

1. Using tags. Make an official tag “TIMECODE” to store a string of the timecode expression that correlates to the Matroska timestamp of 0. This approach is limited to storing timecode that is incremental and continuous. Defaults for the incrementation and the behavior of the timecode could be described as part of the tag definitions and feasibly another tag or tags could be reserved for timecode flags when the timecode behavior is not a default. Defining a use of a “TIMECODE” tag would provide the benefit of standardizing an existing practice in FFmpeg’s Matroska muxer, where rewrapping from a timecode-supportive format to Matroska carries the timecode value over as a tag.

2. Using BlockAdditions as described above. This adds more overhead as each timecode value would be written, but would support non-continuous timecode. In FFmpeg, timecode is sometimes carried as side-data such as from the decklink device via https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0946c0ec177dc48ef0677f890aa42d95e667c417 <https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0946c0ec177dc48ef0677f890aa42d95e667c417>. I’d be interested if the Block Addition Mapping described above could be used to carry the timecode side data from the decklink device (or other incoming stream with timecode side data) and store the result within the written Matroska BlockGroups.

Comments welcome.

Thanks much,
Dave Rice