Re: [Cellar] CodecIDs for non-standard and unregistered data formats

[Matt Gruenke <[e-mail]>]

On Wed, Apr 26, 2017 at 5:13 AM, Steve Lhomme <[e-mail]> wrote:

> 2017-02-27 4:04 GMT+01:00 Matt Gruenke <[e-mail]>:
> > Let's take this example of a camera having both a visible light sensor
> and a
> > depth sensor, as well as a stereo microphone.  For various reasons, its
> > depth sensor captures frames at different times and a different rate than
> > its color image sensor.  I'd like to record a video file containing an
> audio
> > track, a normal video track, and a depth track.  I'd like the audio and
> > video portions to be playable in normal video player apps.  When using
> > enhanced apps, the depth data may also be utilized.
>
> It seems that the depth data cannot be used on their own and would
> need to be coupled with the video frames. So that would probably go as
> a BlockAddition data (or something similar as it's currently tied to
> the Track codec). Even if they are repeated for each video frame which
> the sensor doesn't change the value.
>

I'd say it's no more or less coupled than audio.  If a movie has an audio
track, people will typically watch the video with the audio, but sometimes
without.  They rarely listen to the audio without the video, but they could
(and some do).  I think it's a good analogy for depth - it's an optional
augmentation of the video and could be paired with one of several different
video tracks (e.g. if the same file has a depth track and video from both a
visible light & a thermal sensor).

More importantly, if the depth frames are taken at slightly different times
than video, it's critical they have timestamps which reflect this.
Furthermore, it would be a really problematic to repeat the same data on
subsequent frames, in cases where no new depth frames have been captured in
between.  These kinds of sins some people committed with video created
substantial headaches for video processing.

Depth must be synchronized with video as precisely as possible.  Any
fudging of timestamps or duplication of data will interfere with attempts
to interpolate the depth frames to match the video (or vice versa).  It
might not be feasible or desirable to perform this interpolation at capture
time.


Matt