Re: [Cellar] AV1 mapping Matroska

[Hendrik Leppkes <[e-mail]>]

On Mon, Jul 2, 2018 at 1:27 PM Steve Lhomme <[e-mail]> wrote:
>
> 2018-07-02 13:14 GMT+02:00 Jerome Martinez <[e-mail]>:
> > On 02/07/2018 12:51, Steve Lhomme wrote:
> >>
> >> 2018-07-02 12:34 GMT+02:00 Michael Niedermayer <[e-mail]>:
> >>>
> >>> Hi
> >>>
> >>> On Mon, Jul 02, 2018 at 09:47:45AM +0200, Moritz Bunkus wrote:
> >>>>
> >>>> Hey,
> >>>>
> >>>>> Looking at the term "coded video sequence" (CVS) in other codecs (H.264
> >>>>> and H.265) it seems it's a common term. And for those codec one Segment
> >>>>> correspond to one CVS, with the parameters of that CVS stored in the
> >>>>> CodecPrivate (SPS + PPS for H.264 for example). So we should probably
> >>>>> go
> >>>>> in that simple way.
> >>>>
> >>>> I have quite a lot of h.264 samples here, mainly M2TS from DVB, where
> >>>> SPS/PPS change mid-stream, often multiple times. For such files the
> >>>> first
> >>>> occurrences of SPS/PPS make up the AvcC in CodecPrivate, but all key
> >>>> frames
> >>>> are still prefixed with the currently active SPS/PPS — because that's
> >>>> the
> >>>> only way to signal that stuff has actually changed.
> >>>
> >>> IMHO All SPS/PPS should be in the "global header" (CodecPrivate) which
> >>> the
> >>> global header applies to, not just the first.
> >
> >
> > Not possible with e.g. real time streaming (you don't know in advance the
> > next SPS/PPS)
> >
> >>> This way an application can know all resolutions, aspect ratios and so on
> >>> from just the header without the need to scan through all random access
> >>> points.
> >>
> >> My understanding is that it's how it's done in MP4:
> >>
> >> "Each AVC sample entry, which contains the AVC video stream decoder
> >> specific information, includes a group of SPSs and PPSs. This group of
> >> parameter sets functions much like a codebook. Each parameter set has
> >> an identifier, and each slice references the parameter set it was
> >> coded against using the parameter set's identifier."
> >>
> >> I have to check but I don't think Sequence Headers OBUs have
> >> identifiers like that in AV1.
> >
> >
> > Not in AV1 (I just updated my own code for AV1  1.0.0 spec, still not
> > there).
> > But it actually does not answer the issue because in the example there is
> > usually still only the "set id" of 0 (the new SPS replaces the old one, not
> > a new number).
> >
> > Actually IIRC in HEVC (same for AVC) there are explicitly 2 different codec
> > identifiers:
> > - "hvc1" is for *all* SPS/PPS in the global header (CodecPrivate), and there
> > is no SPS/PPS in the stream itself
> > - "hev1" is for *no* SPS/PPS in the global header (CodecPrivate), and key
> > frames are prefixed with the currently active SPS/PPS
> > (I personally don't see the need of 2 different identifiers, as it quick to
> > see if SPS/PPS is in the file, but there is surely a reason)
> >
> > The issue is not only for AV1, but for all formats permitting to have the
> > init in the stream itself (AVC, HEVC, AV1...).
> >
> > IMO we should not restrict to only one possibility in Matroska spec, as they
> > are for different needs (non real time vs real time), whatever is the format
> > (AVC, HEVC, AV1...).
> >
> >
> >
> >> By the way, this is one mapping for V_AV1. There could be other AV1
> >> mapping in Matroska where there is no CodecPrivate and there could be
> >> multiple coded video sequences in it. But it would require the use of
> >> CodecState and CueCodecState which is currently not an option for
> >> WebM. So for now I will concentrate on a single coded video sequence.
> >> And if needed in the future we can do a different one.
> >
> >
> > If WebM decides that real time AV1 is for them, they may decide to discard
> > CodecPrivate usage and put sequence header in the "Block" (as they currently
> > do with up to date AOM build)
> > So restriction will not be there in practice.
>
> Seeking in such a file would not be possible. I'm not sure it's
> desirable even when doing adaptive streaming with short sequences. We
> will need CodecState and CueCodecState for that. We could define such
> a case (with a different codec ID) for Matroska and WebM could pick
> that from there.
>
> Also while working on the case where the Sequence Header OBU doesn't
> change, I noticed an issue. The Sequence Header OBU can still change
> the [operating_parameters_info] during the lifetime of the stream
> (paragraph 7.5 pointed out by Tim). That means even applying the
> stricter restrictions of the Codec Video Sequence in AV1 means we'd
> still need CodecState and CueCodecState for seeking to work. This
> change is only possible in a specific decoder model (Annex E) when
> [decoder_model_info_present_flag] is 1. When it's 0 then the
> [operating_parameters_info] doesn't apply. So we should probably
> restrict the single Sequence Header OBU to streams where
> [decoder_model_info_present_flag] is 0.
>

In this particular case, doesn't the stream already include in-band
Sequence Header OBUs, and judging from the requirement of some
parameters being allowed to change, I would wager that decoders would
be setup to handle them as well.
So all this comes down to is setting up Cues properly to point to
clean seek points with a Sequence Header + Keyframe, much like a H264
stream with minor SPS/PPS changes would be muxed right now.

Now if an encoder would produce a stream that uses multiple different
Sequence Headers but doesn't regularly repeat the sequence header (ie.
with keyframes), then such a stream would be unseekable in whatever
medium, and that would be the encoders fault.

(PS: re-posted to the list, can someone fix this list to set reply-to headers?)

- Hendrik