Re: [Cellar] AV1 seeking

[Steve Lhomme <[e-mail]>]

Hi,

2018-07-16 0:19 GMT+02:00 Andreas Rheinhardt
<[e-mail]>:
>> In MP4 they have [initial_presentation_delay_minus_one] in the
>> CodecPrivate. I did not understand it so far because it's not found in
>> the AV1 spec. But it seems to guarantee that to read frame 'f' you
>> need to decode X frame before that one. In our case that would be 5 to
>> have at least a decoded.
>>
> You completely misunderstood this field. It is the equivalent of the
> max_num_reorder_frames value from H.264. Let me explain it in MPEG
> terminology (with b-frames) as you probably have way more experience
> with this: Consider a decoder that can only decode one frame per unit of
> time and a stream like this (left to right is decoding order; the
> numbers are presentation order):
> I0 P2 B1 ...
> If one displayed the leading I frame immediately after decoding it, one
> would not have the right frame to display at time 1, because at that
> time only I0 and P2 has been decoded, not B1. Therefore one has to
> decode I0 and P2 before one outputs the first frame and and
> max_num_reorder_frames would be 1. The typical b-pyramid would require
> to decode the first three frames before the display of the first frame
> and max_num_reorder_frames would be 2.
> The [initial_presentation_delay_minus_one] is the AV1 analogue of this.
> This number is not an upper bound for the amount of temporal units
> between delayed RAP and recovery point/for the amount of frames shared
> between a GOP. Just look at this example:
> MPEG example
> I0 P5 B1 B2 B3 B4 I10 B6 B7 B8 B9
> AV1 example (I kept the MPEG-naming with P and B to make it easier
> comparable to the above; furthermore the pointer *x denotes that a frame
> is showable and x without * is a frame header that outputs *x via
> show_existing_frame;  square brackets are the delimiters of temporal
> units; I10 is the delayed RAP frame)
> [I0] [*P5 B1] [B2] [B3] [B4] [P5] [*I10 B6] [B7] [B8] [B9] [I10] ...
> This stream can have [initial_presentation_delay_minus_one] equal to 1,

A value of 1 means 2 frame delay. That would mean B6 needs *I10 and P5 ?

> yet in order to seek to [I10] one has to decode the temporal unit [*I10
> B6] (or at least the decodable keyframe in it) which is four temporal
> units in front of [I10].

That seems more like it.

In the Frame presentation timing paragraph (E.4.7) it says:

InitialPresentationDelay =  Removal [ initial_display_delay_minus_1 ]
+ TimeToDecode [ initial_display_delay_minus_1 ]

and

PresentationTime[ 0 ] = InitialPresentationDelay
PresentationTime[ j ] = InitialPresentationDelay + (
frame_presentation_time[ j ] - frame_presentation_time[ 0 ] ) * DispCT

or in constant bitrate mode

PresentationTime[ 0 ] = InitialPresentationDelay
PresentationTime[ j ] = PresentationTime[ j - 1 ] + (
num_ticks_per_picture_minus_1 + 1 ) * DispCT


And our `Block` timestamp derives directly from this
[PresentationTime]. The delay is carried over every frame. So it does
seem like we need to globally shift these timestamps for the Track.
Possibly with `CodecDelay`.

Basically each frame has its timestamp on which this delay has to be
added (can't be negative). While `CodecDelay` is a positive value that
needs to be substracted from the timestamps. It is available in WebM,
because Opus needs it.

Let's assume CodecDelay = InitialPresentationDelay

The `Block` timestamps would be stored like this:
Block[ 0 ] = 0
Block[ 1 ] = frame_presentation_time[ 1 ] * DispCT
Block[ 2 ] = frame_presentation_time[ 2 ] * DispCT
...

And on output of the demuxer we would get:
Block[ 0 ] = -InitialPresentationDelay
Block[ 1 ] = -InitialPresentationDelay + frame_presentation_time[ 1 ] * DispCT
Block[ 2 ] = -InitialPresentationDelay + frame_presentation_time[ 2 ] * DispCT
...

`CodecDelay` cannot be used here. But it's very close to what we need.

It's not the SeekPreRoll either because the actual frame timestamps of
each frame is affected, not just when seeking (although it may be
needed for delayed RAP).

We also need to figure out whether the Blocks need to be stored with
or without the delay. If that's with the delay then we don't even need
to care about it. The frames will just not start at 0 but that's
already the case in the original stream.

But that may just be the tricky part here, the reference point. The
PresentationTime[ 0 ] doesn't start at 0 because some frames were
needed to decode before getting actual usable data out of the decoder.
But this is really the first frame to display so it would actually be
0 on the output of the demuxer. So it does seem exactly like what
CodecDelay does:
"CodecDelay is The codec-built-in delay in nanoseconds. This value
must be subtracted from each block timestamp in order to get the
actual timestamp."

The `Block` timestamp would be PresentationTime[ 0 ] which isn't 0.
But on the output Block[ 0 ] should really give 0. So we would have
something like this in the file:
Block[ 0 ] = InitialPresentationDelay
Block[ 1 ] = InitialPresentationDelay + frame_presentation_time[ 1 ] * DispCT
Block[ 2 ] = InitialPresentationDelay + frame_presentation_time[ 2 ] * DispCT
...

And on the output of the demuxer we would have the following, when
`CodecDelay` is [InitialPresentationDelay]:
Block[ 0 ] = 0
Block[ 1 ] = frame_presentation_time[ 1 ] * DispCT
Block[ 2 ] = frame_presentation_time[ 2 ] * DispCT
...

It does look like what the internal AV1 display delay is trying to achieve.

The `CodecDelay` also has this note on proper muxing:
The value SHOULD be small so the muxing of tracks with the same actual
timestamp are in the same Cluster.

Because muxing is done based on the stored `Block` timestamp. It
usually doesn't take in account CodecDelay (but it could/should).

So I'll update my AV1 codec mapping saying `CodecDelay` is
[InitialPresentationDelay] and how to use it.