Re: [Cellar] Ancillary data in Matroska

[Dave Rice <[e-mail]>]

Hi all,

> On Dec 27, 2016, at 3:22 PM, Jerome Martinez <[e-mail]> wrote:
> 
> We need additional codec to be supported in Matroska in order to be able to transwrap in a lossless manner content from older containers to Matroska, especially time codes (all the codecs defined in the patch can contain 1 or more time codes)
> The global idea is to be able to transwrap e.g. MOV (from Apple), MXF (from SMPTE), GXF (from Grass Valley, also known as SMPTE ST 360 and SMPTE RDD 14, found in some broadcaster archives), LXF (from Leitch/Harris, found in some broadcaster archives) without losing metadata in some sidecar non visible streams.

I think this is an interesting approach and it’s probably better than yet again defined another timecode containment system.

> After agreeing on the format, we will have to update tools e.g. FFmpeg in order to support such lossless transwrap.
> 
> 
> I added an "Ancillary" section with several mappings in the attached patch, please comment.
> 
> 
> Comments:
> 
> The frequent remark is "Why do you need so many time code formats, choose one and stop using the others": in the real world, people consider that one format is the best one, but not the same format as others, and they use only this time code for their work and don't plan to switch. You can have 10+ time codes in a file and each one is important for someone, as well as the sidecar information transported at the same time (never same between each format). There is no consensus about that for years, and I don't think we could achieve a consensus from the different actors (lot of them are not interested by our work at IETF) so the idea here is to be able to move from one container to another one without losing any metadata, absolutely not deciding about a reference time code (and other metadata) format, as Matroska does for e.g. video (Matroska does not say that VP9 should be used instead of AVC, it supports both).

Agreed.

> "N_" prefix arbitrary used (2nd letter of "Ancillary" because first letter is already used)

Please also add a patch for the TrackType Element in ebml_matroska.xml as we’ll need an unsigned integer to represent the ‘ancillary’ type of track.

> N_QUICKTIME: is a pure copy of V_QUICKTIME and A_QUICKTIME already supported in Matroska, N_ would be used which would be used for e.g. tmcd codec (QuickTime time code). Exactly same principles as video and audio. Note that classic usage is to store only the first frame content and other content is computed from it, so a player would have to read the first frame even if there is a direct seek request to another place; this could be avoided with some "hack" in the track header, could be the next step after this one is accepted.

Would need a clear caveat in the definition that this implementation would only support storage of the first timecode value and computation of each additional value. A transwrap from a mov file with a timecode track that uses an edit list to handle non-continuous timecode could not be a lossless transwrap in this case, since the edit list would be lost and the new timecode track in the Matroska file would appear continuous.

Also reference to tmcd atom is not clear enough. Is it tref/tmcd, gmhd/tmcd, stsd/tmcd.

If this is stsd/tmcd (I presume), then what time scale should be used? The time scale stored in tmcd or the time scale of the Matroska track?

I like that copying stsd/tmcd copies in the timecode label as well.

> VBI and ANC: used to transport time codes (LTC, VITC, ATC...), Recording Information, bar and pan/scan data, captions (North American CEA-608, CEA-708, European and Australian WSS/Teletext, Japanese ARIB B37...), camera acquisition dynamic metadata, Audio Metadata, Film Transfer and Video Production Information (in theory, I never saw it)...
> The "perfect" solution would be to decode VBI and ANC, and put each format in its own specification, but it would be an huge task and we still would have to store opaque content (content is present in the VBI/ANC but we ignore all from it right now), so it is better to start by the beginning and we consider all opaque without trying to decode the data, as do other containers.

I agree to store and consider it as opaque or to leave within the video but use the PixelCrop elements to note that the lines of the VBI data aren’t intended in presentation. 

> +**Codec ID:** N_VBI  
> +**Codec Name:** Vertical Blanking Interval  
> +**Description:** used in SDTV, see https://en.wikipedia.org/wiki/Vertical_blanking_interval for more information. Each Matroska block contains a SMPTE ST 436 VBI Frame Element.  

OK in the patch you reference storage of VBI as smpte 436m frame elements (example attached to https://trac.ffmpeg.org/ticket/726). Is the N_ prefix still relevant as the encoding would contain captioning data as well? “S_”?

> N_STRIPPEDTIMECODE: the Cluster part will contain no data, but despite of that it must be a complete standalone track, with Track Id ans so on, as it is a standalone track in the source container.

Do you mean that there would be an N_STRIPPEDTIMECODE but it would reference no Clusters, or that the N_STRIPPEDTIMECODE track would reference Clusters that contain no SimpleBlock and no BlockGroup? If the latter where would the value of the timecode be stored?

> +**Codec ID:** N_STRIPPEDTIMECODE  
> +**Codec Name:** Stripped time code  
> +**Description:** track containing no data in the Cluster elements, as all is in the track header: the CodecPrivate element contains, in big endian: 4 bytes Rate numerator, 4 bytes rate denominator, 4 bytes count of consecutive time codes / frames (the "duration", 0 if unknown), 1 byte time code drop frame flag (0 is no, 1 is yes, others are reserved), 4 byte the start time code (in frames). Note: this permits to map a MXF time code track or a GXF stripped time code track to Matroska.  

“no data in the Cluster elements” is not clear enough. The structure of CodecPrivate would not note flags such as if the timecode is limited to 24 hours or not. If that’s similar to the MXF and GXF limitations then I guess it’s fine. Any reference to the CodecPrivate structure? From the writing it’s not clear if this is a new structure or copied from some other timecode storage form.

I suggest throughout the patch we are consistent on capitalization (Frame vs frame) and spacing (timecode vs time code).

Also I’m worried about the confusion to be created between the Matroska Timecode* Elements and the concept of timecode (copies to external reference systems from source objects). An earlier discussion suggested calling the focus of this patch as Reference Timecode to distinguish it from Matroska’s own Timecode. I see that SMPTE is starting to reference timecode as “Time Label” which may also be appropriate here.

> N_SDTICP: nearly 1:1 copy of SMPTE ST 385 elements. In order to have all data and being able to have only 1 Matroska block for all STDI-CP elements, I kept also the 2 last bytes of the KLV "Key" and the KLV "Length" in addition of the KLV "Value". Note that something weird, this is a standalone track in source container but the difference with other tracks (including ANC or VBI) is that this stream has no track header so no track number, and Matroska specs makes TrackNumber mandatory, I wonder what should be used during transwrap (fake number?)

Not familiar with this type of timecode. Would max-TrackNumber + 1 suffice?

Dave Rice