Re: [Cellar] QuickTime timecode tracks in Matroska, was Ancillary data in Matroska

[Dave Rice <[e-mail]>]

> On Apr 26, 2017, at 3:51 AM, Steve Lhomme <[e-mail]> wrote:
> 
> A bit late on the party but I agree with the move to a more specific
> track type for timecodes rather than the generic ancillary data.
> 
> 2017-03-20 22:07 GMT+01:00 Jerome Martinez <[e-mail]>:
>> Le 20/03/2017 à 21:37, Dave Rice a écrit :
>>> 
>>> [...]
>>>>> 
>>>>> I’ve only seen a single value stored but then sometimes an edit list
>>>>> used to alter the ordering of the timecode (in case when its
>>>>> non-sequential). For Matroska possibly it could simply store a new timecode
>>>>> Block at any case when it is not sequential.
>>>> 
>>>> What about seeking?
>>> 
>>> Isn't the seeking similar to QuickTime.
>> 
>> 
>> Difference is that in QuickTime, a wrong "seek table" makes the playback
>> impossible, so developers are more careful about it. and in the file I
>> remember (I must still find it), there was only one offset in the offset
>> table of QuickTime and a single "chunk" of time code with byte size of 4
>> bytes per frame (to be confirmed as you say you saw edit lists instead)
>> 
>>> In QuickTime you'd need the offset table of the timecode track to
>>> understand the number and location of all timecode samples. In Matroska you
>>> would use the associated CuePoints of the timecode track for the same
>>> reference. By parsing Cues the reader should be able to determine if the
>>> timecode is stripped or not and if not then have the CueTimes for each
>>> associated timecode Cluster.
>>> 
>>>> If you have a timecode block at the first frame and also at the middle of
>>>> the file, how a player can know that there is a timecode block breaking the
>>>> sequential order without parsing the whole file when there is a seek request
>>>> to the last minute of content? In that case, time code real value would be
>>>> set to "waiting for new block" (and this block never comes). I don't think
>>>> that forcing full parsing of the file with time code for seeking is a good
>>>> solution.
>>> 
>>> Same. But in the case of video if you seek to a P-frame then you have to
>>> go backwards and decode from the I-frame. Similarly if seeking to a
>>> timepoint without a timecode Cluster (analogous to a P-frame), then the
>>> reader would have to use the Cues to decode the prior timecode Cluster
>>> (analogous to I-frame) in order to determine the timecode value of the seek
>>> point.
>> 
>> 
>> Possible.
>> But we need to be careful, what does it means?
>> examples of issue:
>> 1/ can a player consider that if there is only one CuePoint with CueTrack =
>> the ID of the time code track, it means that this is a stripped content and
>> time codes are in sequential order?
>> 2/ can a player consider that if there are only 2 CuePoints with CueTrack =
>> the ID of the time code track, it means that this is a stripped content and
>> time codes are in sequential order except for one place and there is a
>> single discontinuity?
>> 3/ or must a player consider that if there only 2 CuePoints (let say frame 0
>> and frame 1000) with CueTrack = the ID of the time code track, it means that
>> it must seek to 0 if the requested frame is 999 (so a loooooong read of the
>> file on disk before being able to play frame 999)
>> If answers are 2/ yes 3/ no, I understand that the only method for storing a
>> time code track with all values not sequential is to have a CuePoint for
>> each time code frame (which makes the Cues huge, 15-20 bytes of CuePoint per
>> time code frame).
>> 
>> Please provide example about how you imagine CuePoints in the cases:
>> 1/ sequential time codes during 2000 frames
>> 2/ sequential time codes during 2000 frames except one discontinuity at
>> frame 1000
>> 3/ 2000 different times codes
>> and where a player must seek if the request is to seek to frame 999.
> 
> Are timecode frames supposed to be matching a specific video frame or
> audio frame ? Or they are loose ? If the former case it may be best to
> be attached as a BlockAdditional. Otherwise they should always be in
> sequential/display order, just like audio is. Even if video is not.

I'd propose to limit the use case to the former. The idea of storing timecode data within BlockAdditional is interesting, but would that limit the metadata possibilities. For instance if timecode data is stored as it's own TrackEntry, then potentially a file may contain two timecode TrackEntries, one depicting timecode values from the original source materials used (likely non-continuous), whereas another TrackEntry could represent a new continuous timecode track that served some purpose in a video production. With these timecodes as two TrackEntries with their own Clusters, they could have distinct names and could be independently described with flags, so one may be enabled and default while the other is disabled.

With timecode in BlockAddition, there could potentially be two sets of timecode (two BlockMore), but they would be indistinguishable and could not be controlled or labelled independently.

> Having a separate track means the timestamps may not match exactly the
> video or audio tracks. So in case of seeking/discontinuity it's like
> you said, there's a state where you need to way for the next Timecode
> block to know where you are.

This is the case if the there's one timecode value stored per video frame, but this may not be the case. We also discussed using QuickTime's strategy of storing an integer as a timecode while the timecode track's private data notes what equation to use to convert the integer into a timecode (flags like max24, timebase, etc). In this case a timecode value would only need to be stored at a point in time when the timecode is non-continuous.

> But in general a Timecode block
> should/must be placed before the audio/video it corresponds to. It's
> the same requirement there is in WebM that audio blocks are muxed
> before video blocks. Timecode would be even more prioritary.

Does Matroska have the same ordering requirement? I don't see them. Is it valid to store Clusters in reverse timestamp order?

Dave Rice