Re: [Cellar] AV1 mapping update
"Timothy B. Terriberry" <tterribe@xiph.org> Mon, 16 July 2018 15:11 UTC
Return-Path: <tterribe@xiph.org>
X-Original-To: cellar@ietfa.amsl.com
Delivered-To: cellar@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8498C130EB7 for <cellar@ietfa.amsl.com>; Mon, 16 Jul 2018 08:11:42 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.901
X-Spam-Level:
X-Spam-Status: No, score=-6.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_HI=-5, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PjJri1eMWIFH for <cellar@ietfa.amsl.com>; Mon, 16 Jul 2018 08:11:40 -0700 (PDT)
Received: from smtp.mozilla.org (mx2.scl3.mozilla.com [63.245.214.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 281571310C5 for <cellar@ietf.org>; Mon, 16 Jul 2018 08:11:40 -0700 (PDT)
Received: from localhost (localhost6.localdomain [127.0.0.1]) by mx2.mail.scl3.mozilla.com (Postfix) with ESMTP id E56FDC0A44 for <cellar@ietf.org>; Mon, 16 Jul 2018 15:11:39 +0000 (UTC)
X-Virus-Scanned: amavisd-new at mozilla.org
Received: from smtp.mozilla.org ([127.0.0.1]) by localhost (mx2.mail.scl3.mozilla.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CxNrOYNKkvIE for <cellar@ietf.org>; Mon, 16 Jul 2018 15:11:38 +0000 (UTC)
Received: from [31.133.142.81] (dhcp-8e51.meeting.ietf.org [31.133.142.81]) (Authenticated sender: tterriberry@mozilla.com) by mx2.mail.scl3.mozilla.com (Postfix) with ESMTPSA id 42B80C0654 for <cellar@ietf.org>; Mon, 16 Jul 2018 15:11:38 +0000 (UTC)
To: cellar@ietf.org
References: <CAOXsMFKHo6RS+q8KCXKoKCiBBS9pVqs92wsLgSfXZO+DT3dStQ@mail.gmail.com> <ca0f009e-a245-fcd6-95f8-f051736c9161@googlemail.com> <CAOXsMFL5-MaHQaAOyh7jSFUpCNbSEvAWKmAHcepaF+QsQuYbHw@mail.gmail.com> <fee747da-77ca-9282-a4c3-c112fd746507@googlemail.com> <CAOXsMFJtc9pq+PphRb5kF9Mp4jyS5j3LQi6vQQmHRyTDYWyQ-A@mail.gmail.com> <b8486fa4-132b-f814-7046-91efb0a48ec6@googlemail.com> <10d56d2a-3053-6069-7805-54bb4fd1d4e0@xiph.org> <b8c8236c-5dd7-5511-6994-730ea0aaef84@googlemail.com>
From: "Timothy B. Terriberry" <tterribe@xiph.org>
Message-ID: <c50475ce-3f09-5bf7-32ee-76ee7d55af52@xiph.org>
Date: Mon, 16 Jul 2018 08:11:37 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 SeaMonkey/2.49.7.0
MIME-Version: 1.0
In-Reply-To: <b8c8236c-5dd7-5511-6994-730ea0aaef84@googlemail.com>
Content-Type: text/plain; charset="UTF-8"; format="flowed"
Content-Transfer-Encoding: 7bit
Archived-At: <https://mailarchive.ietf.org/arch/msg/cellar/NXjgaDKZYKNGlEGmrBtdVy-Ep2g>
Subject: Re: [Cellar] AV1 mapping update
X-BeenThere: cellar@ietf.org
X-Mailman-Version: 2.1.27
Precedence: list
List-Id: Codec Encoding for LossLess Archiving and Realtime transmission <cellar.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/cellar>, <mailto:cellar-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/cellar/>
List-Post: <mailto:cellar@ietf.org>
List-Help: <mailto:cellar-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/cellar>, <mailto:cellar-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 16 Jul 2018 15:11:45 -0000
Andreas Rheinhardt wrote: > every timestamp in Matroska is a multiple of 1 ns (this number is > hard-coded) so that NTSC timings can't be exactly represented on the > container level: E.g. the default duration field for 24/1.001fps content > would indicate 41708333ns, although it is 41708333.33333333...ns. So by > preserving the bitstream information it is possible to increase the > certainity that 41708333ns actually means 24/1.001 fps. This seems like a good enough reason that it should probably be explicitly described in the draft (especially if you want any of that to work in practice). If that's the only reason then okay, but if there are a lot of other reasons, then SHOULD may be too strong. As an aside the timestamp situation still makes me sad. > a) I don't like restrictions on the content that can be put into > Matroska either. The reason why I put it in is because this is (in my > understanding) currently an absolute requirement of Matroska > (`PixelWidth` and `PixelHeight` are mandatory and valid for every > frame). Given that decoders will have to support these varying output > dimensions anyway (it is a mandatory part of AV1, isn't it?), it would > be possible to drop this requirement, but for this the specifications of > both Matroska and Webm would have to be changed. I'm not sure if > everyone is ok with this. I don't think this is too burdensome. A system that requires a constant pixel size for the decoded frame can put an upscaler between the underlying AV1 decoder implementation and the input to that system, and pretend that rescaling is part of the AV1 decoder black box (implementing an upscaler is vastly simpler than implementing a whole video codec). The fact that there are different dimensions in the AV1 frame header shouldn't be something it has to worry about. > decoding process, but only internally, so that we can ignore it). Is > that correct? (I have to admit I thought for a time that the output I believe that is correct. The UpscaledWidth allows the codec to partially do what I was describing for real-time/low-latency streams internally, but still run some of the loop filters at the upscaled resolution, but due to hardware decoder complexity concerns it can only rescale in the horizontal direction, so there's a limited operating space where that is sufficient. > is the last frame in that temporal unit. I don't see a reason why there > should be another frame afterwards in the same temporal unit, but is it > actually allowed? I don't know. I agree it's ambiguous as written. I'll ask someone and get back to you. > 2. Would it be legal to code a frame with [show_frame] 0, > [showable_frame] 1 followed (perhaps immediately) by a frame header in > the same temporal unit that has [show_existing_frame] set to 1 and > outputs the frame just coded? If yes, is there any reason to ever use > it? I don't see any. According to the definition, this temporal unit I think it is legal. I don't know why you would ever do it, but it wouldn't cause any great harm. Does it complicate anything to allow it any more than DRAPs complicate things already? > 3. Would it be legal to have a temporal unit that contains a frame that > is not shown (but might be declared as showable) followed by a keyframe > with [show_frame] equal to 1? I'm asking because both the current > proposal as well as the mp4/ISOBMFF definition of sync sample require > the keyframe to be the first frame in the temporal unit. I think this is legal. I also think this would be entirely wasteful. The key frame would reset all of the reference buffers, so the non-shown frame would never contribute anything to a shown frame. > 4. You have only criticized the "MUST only" wording of the definition of > keyframe simpleblocks. You said nothing about the requirement that it is > the first frame that needs to be of type KEY_FRAME. The standard doesn't Please assume that if I don't comment on something, it may just be because I have not thought it all of the way through (you might not be wrong to assume that about things I do comment on, also). > overwritten by the delayed RAP frame). Then it may make sense to first > code this frame as a showable frame using all the reference frames and > then code the delayed RAP frame followed by the frame that is actually > output for the temporal unit containing the delayed RAP. Is my usecase > sound and should the "first"-requirement be dropped? I agree that, because a DRAP does not immediately reset all of the reference buffers, it may not be wasteful to precede it by a non-shown frame, and because you will (presumably) consume at least one reference buffer slot to store the DRAP, you cannot in general re-order them. It gets more complicated when you consider spatial scalability. It is possible for you to have a RAP for a frame other than the base layer, so you may actually have one or more shown frames in the same temporal unit (for different spatial layers) before a (delayed or even non-delayed) RAP, and again be unable to re-order them, and now whether you can start display at that frame or a following frame can depend on which operating point you are trying to use. In particular, the problem is that the AV1 specification defines a RAP in terms of a *frame*, and not in terms of a temporal unit. Because Matroska defines its blocks as containing a temporal unit, and RAPs in terms of a block, you will have to define the details of when a temporal unit should be considered a RAP (or DRAP, KFDRP, etc.) based on the frames contained inside of it. I think it may have been a mistake to do things this way in the AV1 specification, since the idea was always for temporal units to be the unit of muxing in any container, but I expect it was done this way out of expediency to avoid having to think through all of these cases. We have to think through them now. As I said in another mail, I think it may be okay to impose additional restrictions on top of the AV1 specification, especially if it makes everyone else's lives easier. This may be one such case. While the situations described above may confer some marginal advantage, and I can imagine an encoder that could take advantage of them, I don't think there would be a big loss by forbidding them. I do think it would be a very good idea to have the *same* restrictions (if any) for Matroska and MP4. If something is technically allowed by the AV1 specification but you can't actually put that into any container, then in practical terms it might as well not be allowed. But if we allow some things in Matroska but don't allow them in MP4, or vice versa, then you just have an interoperability mess. > 5. What do you think of the restriction of a Matroska track to a (subset > of) a CVS? If you look back to my mail on June 30th, I thought I was the one who suggested it in the first place.
- [Cellar] AV1 mapping update Steve Lhomme
- Re: [Cellar] AV1 mapping update Andreas Rheinhardt
- Re: [Cellar] AV1 mapping update Steve Lhomme
- Re: [Cellar] AV1 mapping update Steve Lhomme
- Re: [Cellar] AV1 mapping update Andreas Rheinhardt
- Re: [Cellar] AV1 mapping update Steve Lhomme
- Re: [Cellar] AV1 mapping update Steve Lhomme
- Re: [Cellar] AV1 mapping update Timothy B. Terriberry
- Re: [Cellar] AV1 mapping update Andreas Rheinhardt
- Re: [Cellar] AV1 mapping update Timothy B. Terriberry
- Re: [Cellar] AV1 mapping update Andreas Rheinhardt
- Re: [Cellar] AV1 mapping update Timothy B. Terriberry
- Re: [Cellar] AV1 mapping update Timothy B. Terriberry
- Re: [Cellar] AV1 mapping update Steve Lhomme