Re: [Cellar] AV1 mapping Matroska

[Steve Lhomme <[e-mail]>]

2018-07-02 13:35 GMT+02:00 Hendrik Leppkes <[e-mail]>:
> On Mon, Jul 2, 2018 at 1:27 PM Steve Lhomme <[e-mail]> wrote:
>>
>> 2018-07-02 13:14 GMT+02:00 Jerome Martinez <[e-mail]>:
>> > On 02/07/2018 12:51, Steve Lhomme wrote:
>> >>
>> >> 2018-07-02 12:34 GMT+02:00 Michael Niedermayer <[e-mail]>:
>> >>>
>> >>> Hi
>> >>>
>> >>> On Mon, Jul 02, 2018 at 09:47:45AM +0200, Moritz Bunkus wrote:
>> >>>>
>> >>>> Hey,
>> >>>>
>> >>>>> Looking at the term "coded video sequence" (CVS) in other codecs (H.264
>> >>>>> and H.265) it seems it's a common term. And for those codec one Segment
>> >>>>> correspond to one CVS, with the parameters of that CVS stored in the
>> >>>>> CodecPrivate (SPS + PPS for H.264 for example). So we should probably
>> >>>>> go
>> >>>>> in that simple way.
>> >>>>
>> >>>> I have quite a lot of h.264 samples here, mainly M2TS from DVB, where
>> >>>> SPS/PPS change mid-stream, often multiple times. For such files the
>> >>>> first
>> >>>> occurrences of SPS/PPS make up the AvcC in CodecPrivate, but all key
>> >>>> frames
>> >>>> are still prefixed with the currently active SPS/PPS — because that's
>> >>>> the
>> >>>> only way to signal that stuff has actually changed.
>> >>>
>> >>> IMHO All SPS/PPS should be in the "global header" (CodecPrivate) which
>> >>> the
>> >>> global header applies to, not just the first.
>> >
>> >
>> > Not possible with e.g. real time streaming (you don't know in advance the
>> > next SPS/PPS)
>> >
>> >>> This way an application can know all resolutions, aspect ratios and so on
>> >>> from just the header without the need to scan through all random access
>> >>> points.
>> >>
>> >> My understanding is that it's how it's done in MP4:
>> >>
>> >> "Each AVC sample entry, which contains the AVC video stream decoder
>> >> specific information, includes a group of SPSs and PPSs. This group of
>> >> parameter sets functions much like a codebook. Each parameter set has
>> >> an identifier, and each slice references the parameter set it was
>> >> coded against using the parameter set's identifier."
>> >>
>> >> I have to check but I don't think Sequence Headers OBUs have
>> >> identifiers like that in AV1.
>> >
>> >
>> > Not in AV1 (I just updated my own code for AV1  1.0.0 spec, still not
>> > there).
>> > But it actually does not answer the issue because in the example there is
>> > usually still only the "set id" of 0 (the new SPS replaces the old one, not
>> > a new number).
>> >
>> > Actually IIRC in HEVC (same for AVC) there are explicitly 2 different codec
>> > identifiers:
>> > - "hvc1" is for *all* SPS/PPS in the global header (CodecPrivate), and there
>> > is no SPS/PPS in the stream itself
>> > - "hev1" is for *no* SPS/PPS in the global header (CodecPrivate), and key
>> > frames are prefixed with the currently active SPS/PPS
>> > (I personally don't see the need of 2 different identifiers, as it quick to
>> > see if SPS/PPS is in the file, but there is surely a reason)
>> >
>> > The issue is not only for AV1, but for all formats permitting to have the
>> > init in the stream itself (AVC, HEVC, AV1...).
>> >
>> > IMO we should not restrict to only one possibility in Matroska spec, as they
>> > are for different needs (non real time vs real time), whatever is the format
>> > (AVC, HEVC, AV1...).
>> >
>> >
>> >
>> >> By the way, this is one mapping for V_AV1. There could be other AV1
>> >> mapping in Matroska where there is no CodecPrivate and there could be
>> >> multiple coded video sequences in it. But it would require the use of
>> >> CodecState and CueCodecState which is currently not an option for
>> >> WebM. So for now I will concentrate on a single coded video sequence.
>> >> And if needed in the future we can do a different one.
>> >
>> >
>> > If WebM decides that real time AV1 is for them, they may decide to discard
>> > CodecPrivate usage and put sequence header in the "Block" (as they currently
>> > do with up to date AOM build)
>> > So restriction will not be there in practice.
>>
>> Seeking in such a file would not be possible. I'm not sure it's
>> desirable even when doing adaptive streaming with short sequences. We
>> will need CodecState and CueCodecState for that. We could define such
>> a case (with a different codec ID) for Matroska and WebM could pick
>> that from there.
>>
>> Also while working on the case where the Sequence Header OBU doesn't
>> change, I noticed an issue. The Sequence Header OBU can still change
>> the [operating_parameters_info] during the lifetime of the stream
>> (paragraph 7.5 pointed out by Tim). That means even applying the
>> stricter restrictions of the Codec Video Sequence in AV1 means we'd
>> still need CodecState and CueCodecState for seeking to work. This
>> change is only possible in a specific decoder model (Annex E) when
>> [decoder_model_info_present_flag] is 1. When it's 0 then the
>> [operating_parameters_info] doesn't apply. So we should probably
>> restrict the single Sequence Header OBU to streams where
>> [decoder_model_info_present_flag] is 0.
>>
>
> In this particular case, doesn't the stream already include in-band
> Sequence Header OBUs, and judging from the requirement of some
> parameters being allowed to change, I would wager that decoders would
> be setup to handle them as well.
> So all this comes down to is setting up Cues properly to point to
> clean seek points with a Sequence Header + Keyframe, much like a H264
> stream with minor SPS/PPS changes would be muxed right now.

Ah yes, good point. In Paragraph 7.5 there is this:

"Note:There is not a requirement that every temporal unit with a key
frame also contains a sequence header, just that the sequence header
has been sent before the first key frame. However, note that temporal
units without sequence header OBUs are not considered to be random
access points."

So we can assume that differing Sequence Headers OBUs will be found
in-band. And must be bit-identical to the first one in the CVS except
for [operating_parameters_info].

It may also mean that our keyframe flag may need to be refined. And by
"temporal unit" I think it means "temporal unit with a key frame"
wihout a sequence header OBU are not RAP. In practice I don't think
there will not be keyframes where the sequence header OBU is not in
that temporal unit.


> Now if an encoder would produce a stream that uses multiple different
> Sequence Headers but doesn't regularly repeat the sequence header (ie.
> with keyframes), then such a stream would be unseekable in whatever
> medium, and that would be the encoders fault.
>
> (PS: re-posted to the list, can someone fix this list to set reply-to headers?)
>
> - Hendrik
>
> _______________________________________________
> Cellar mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/cellar



-- 
Steve Lhomme
Matroska association Chairman