Re: [Cellar] AV1 mapping Matroska

Hendrik Leppkes <h.leppkes@gmail.com> Mon, 02 July 2018 11:35 UTC

Return-Path: <h.leppkes@gmail.com>
X-Original-To: cellar@ietfa.amsl.com
Delivered-To: cellar@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 71564130E0A for <cellar@ietfa.amsl.com>; Mon, 2 Jul 2018 04:35:15 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HBjztX7m8KNE for <cellar@ietfa.amsl.com>; Mon, 2 Jul 2018 04:35:13 -0700 (PDT)
Received: from mail-wm0-x234.google.com (mail-wm0-x234.google.com [IPv6:2a00:1450:400c:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0FA20130EBB for <cellar@ietf.org>; Mon, 2 Jul 2018 04:35:11 -0700 (PDT)
Received: by mail-wm0-x234.google.com with SMTP id w16-v6so8007855wmc.2 for <cellar@ietf.org>; Mon, 02 Jul 2018 04:35:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-transfer-encoding; bh=gnyChOSJCxJhT3rs5wTXwJloenfRrdunTxJAk+6CudI=; b=qys7DvImVckbxGOiWnrWh3V5+wxvLGdU6Vun//m1yyX2C+muuTv2/sH7i3pxOYgne2 dcERTPMThJSxy+Djpj9fQBi2+UFjheogktCGIBA8/8VECun1NVlrbfLLIvf4Fd2cfV/9 VJxBzaEUwEgSMl54r4+pmMsbiLFvLcR99DJuTpbCDXhD4+SD+nbaxXDwJn/FiEkEFoB8 MBwZcLHSuiMPBzkA/9hiXB9qTH9s7Zct4NhyMh65g3VNElz+Qrwk+lr0Ttb1DNs7uC8/ XN6bX9nixCDn6R804Tb4um4emI+o2BStQD3EdG/barUa3Fp06UD2CKE4Hrdq+a+fQaby Mafw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=gnyChOSJCxJhT3rs5wTXwJloenfRrdunTxJAk+6CudI=; b=Qdgclau/JtLWVBuNZylaN8zM7kT9Frzpx85YgGCSCq/Wb4wdKSCbqy6cEsHK5cIi5N 6rLfO3Z9Dfv9QZjmdxWQsOQ1Cd36yvl3e8IktfxsKsJxBt3X5IGkc508uztwZQfuaWgl gxkPGQxjXdCMk02qhnDno8Ho8WVyJxI49RcLD8FtlwpSeBIhUAPAG2tn1tf0CfLt+V+W N30CXf3oHywU+ns6xqbierKUpA+GT1JhQT66hpWfBB9uVYDOeibkFH+sZfcKE2o/gSWu NkphKB/yFOZG573Pi5ZGgmQhry2OhSV6J6HHy3pBcCLVAJH1Rir62ZjsljzToMbvKjSj QTfQ==
X-Gm-Message-State: APt69E2WoeEfm6E1UfT0MAL+op19+3icKeBkg6y3b42B1X+5YRleJBje 5IAtnKSOJZDuWbeEu0aGxaO8FG5lkgXjsO2pTaPzdvf3
X-Google-Smtp-Source: AAOMgpfZQ1k/kyKZwh0kl54aC7yV/DZx6v1RBDPQUM4NoMz8EsNhcSvcoGAqGKZEjPQZyDDjfn53BI6Xl+IB5mbkCWw=
X-Received: by 2002:a1c:8d15:: with SMTP id p21-v6mr7357699wmd.145.1530531309319; Mon, 02 Jul 2018 04:35:09 -0700 (PDT)
MIME-Version: 1.0
References: <f603a9f5-d7dc-6640-1f53-f4d7b62a788e@googlemail.com> <CAOXsMFJJbHbh3=Gnu7Uag4Skq5+=jOimFNPxuicgDiVdbven4w@mail.gmail.com> <dada3690-c8da-05db-4ddb-32476c1170f5@xiph.org> <CAOXsMF+CPW6VJsFaeahuL00hnvq5dUO-gHE3sYXPr5iVy8s1aQ@mail.gmail.com> <87efgmdqou.fsf@bunkus.org> <20180702103455.GQ4839@michaelspb> <CAOXsMF+Q-RzwapNOamqc9G9XttnzMXyUuSY8_HFAA+h+4YsMJA@mail.gmail.com> <36a0dc7e-8579-d99c-4beb-e8f4ce019eff@mediaarea.net> <CAOXsMFKwoujXN1zqzDN16dcw5=14-Mer+XoJZZMzPbo+JpS_XA@mail.gmail.com>
In-Reply-To: <CAOXsMFKwoujXN1zqzDN16dcw5=14-Mer+XoJZZMzPbo+JpS_XA@mail.gmail.com>
From: Hendrik Leppkes <h.leppkes@gmail.com>
Date: Mon, 2 Jul 2018 13:35:00 +0200
Message-ID: <CA+anqdzpJH+HkAhLwQn2NmpxRuFKtJazN3W70s4BvVpREqeOXQ@mail.gmail.com>
To: CELLAR list <cellar@ietf.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://mailarchive.ietf.org/arch/msg/cellar/TU9ZAEVKPuquAkI3IVp0A5jml8I>
Subject: Re: [Cellar] AV1 mapping Matroska
X-BeenThere: cellar@ietf.org
X-Mailman-Version: 2.1.26
Precedence: list
List-Id: Codec Encoding for LossLess Archiving and Realtime transmission <cellar.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/cellar>, <mailto:cellar-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/cellar/>
List-Post: <mailto:cellar@ietf.org>
List-Help: <mailto:cellar-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/cellar>, <mailto:cellar-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 02 Jul 2018 11:35:15 -0000

On Mon, Jul 2, 2018 at 1:27 PM Steve Lhomme <slhomme@matroska.org> wrote:
>
> 2018-07-02 13:14 GMT+02:00 Jerome Martinez <jerome@mediaarea.net>et>:
> > On 02/07/2018 12:51, Steve Lhomme wrote:
> >>
> >> 2018-07-02 12:34 GMT+02:00 Michael Niedermayer <michael@niedermayer.cc>cc>:
> >>>
> >>> Hi
> >>>
> >>> On Mon, Jul 02, 2018 at 09:47:45AM +0200, Moritz Bunkus wrote:
> >>>>
> >>>> Hey,
> >>>>
> >>>>> Looking at the term "coded video sequence" (CVS) in other codecs (H.264
> >>>>> and H.265) it seems it's a common term. And for those codec one Segment
> >>>>> correspond to one CVS, with the parameters of that CVS stored in the
> >>>>> CodecPrivate (SPS + PPS for H.264 for example). So we should probably
> >>>>> go
> >>>>> in that simple way.
> >>>>
> >>>> I have quite a lot of h.264 samples here, mainly M2TS from DVB, where
> >>>> SPS/PPS change mid-stream, often multiple times. For such files the
> >>>> first
> >>>> occurrences of SPS/PPS make up the AvcC in CodecPrivate, but all key
> >>>> frames
> >>>> are still prefixed with the currently active SPS/PPS — because that's
> >>>> the
> >>>> only way to signal that stuff has actually changed.
> >>>
> >>> IMHO All SPS/PPS should be in the "global header" (CodecPrivate) which
> >>> the
> >>> global header applies to, not just the first.
> >
> >
> > Not possible with e.g. real time streaming (you don't know in advance the
> > next SPS/PPS)
> >
> >>> This way an application can know all resolutions, aspect ratios and so on
> >>> from just the header without the need to scan through all random access
> >>> points.
> >>
> >> My understanding is that it's how it's done in MP4:
> >>
> >> "Each AVC sample entry, which contains the AVC video stream decoder
> >> specific information, includes a group of SPSs and PPSs. This group of
> >> parameter sets functions much like a codebook. Each parameter set has
> >> an identifier, and each slice references the parameter set it was
> >> coded against using the parameter set's identifier."
> >>
> >> I have to check but I don't think Sequence Headers OBUs have
> >> identifiers like that in AV1.
> >
> >
> > Not in AV1 (I just updated my own code for AV1  1.0.0 spec, still not
> > there).
> > But it actually does not answer the issue because in the example there is
> > usually still only the "set id" of 0 (the new SPS replaces the old one, not
> > a new number).
> >
> > Actually IIRC in HEVC (same for AVC) there are explicitly 2 different codec
> > identifiers:
> > - "hvc1" is for *all* SPS/PPS in the global header (CodecPrivate), and there
> > is no SPS/PPS in the stream itself
> > - "hev1" is for *no* SPS/PPS in the global header (CodecPrivate), and key
> > frames are prefixed with the currently active SPS/PPS
> > (I personally don't see the need of 2 different identifiers, as it quick to
> > see if SPS/PPS is in the file, but there is surely a reason)
> >
> > The issue is not only for AV1, but for all formats permitting to have the
> > init in the stream itself (AVC, HEVC, AV1...).
> >
> > IMO we should not restrict to only one possibility in Matroska spec, as they
> > are for different needs (non real time vs real time), whatever is the format
> > (AVC, HEVC, AV1...).
> >
> >
> >
> >> By the way, this is one mapping for V_AV1. There could be other AV1
> >> mapping in Matroska where there is no CodecPrivate and there could be
> >> multiple coded video sequences in it. But it would require the use of
> >> CodecState and CueCodecState which is currently not an option for
> >> WebM. So for now I will concentrate on a single coded video sequence.
> >> And if needed in the future we can do a different one.
> >
> >
> > If WebM decides that real time AV1 is for them, they may decide to discard
> > CodecPrivate usage and put sequence header in the "Block" (as they currently
> > do with up to date AOM build)
> > So restriction will not be there in practice.
>
> Seeking in such a file would not be possible. I'm not sure it's
> desirable even when doing adaptive streaming with short sequences. We
> will need CodecState and CueCodecState for that. We could define such
> a case (with a different codec ID) for Matroska and WebM could pick
> that from there.
>
> Also while working on the case where the Sequence Header OBU doesn't
> change, I noticed an issue. The Sequence Header OBU can still change
> the [operating_parameters_info] during the lifetime of the stream
> (paragraph 7.5 pointed out by Tim). That means even applying the
> stricter restrictions of the Codec Video Sequence in AV1 means we'd
> still need CodecState and CueCodecState for seeking to work. This
> change is only possible in a specific decoder model (Annex E) when
> [decoder_model_info_present_flag] is 1. When it's 0 then the
> [operating_parameters_info] doesn't apply. So we should probably
> restrict the single Sequence Header OBU to streams where
> [decoder_model_info_present_flag] is 0.
>

In this particular case, doesn't the stream already include in-band
Sequence Header OBUs, and judging from the requirement of some
parameters being allowed to change, I would wager that decoders would
be setup to handle them as well.
So all this comes down to is setting up Cues properly to point to
clean seek points with a Sequence Header + Keyframe, much like a H264
stream with minor SPS/PPS changes would be muxed right now.

Now if an encoder would produce a stream that uses multiple different
Sequence Headers but doesn't regularly repeat the sequence header (ie.
with keyframes), then such a stream would be unseekable in whatever
medium, and that would be the encoders fault.

(PS: re-posted to the list, can someone fix this list to set reply-to headers?)

- Hendrik