Re: [AVTCORE] [External] VVC payload format, open issue cluster #1 (SDP for scalability)

Stephan Wenger <stewe@stewe.org> Fri, 18 December 2020 21:55 UTC

Return-Path: <stewe@stewe.org>
X-Original-To: avt@ietfa.amsl.com
Delivered-To: avt@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C83DC3A0990 for <avt@ietfa.amsl.com>; Fri, 18 Dec 2020 13:55:39 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Level:
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=steweorg.onmicrosoft.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aWiyOp_ebrCL for <avt@ietfa.amsl.com>; Fri, 18 Dec 2020 13:55:36 -0800 (PST)
Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2138.outbound.protection.outlook.com [40.107.236.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8C48C3A097E for <avt@ietf.org>; Fri, 18 Dec 2020 13:55:36 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I/2IDM8HQtWWE70QjceQbVPHSr3+nDNWIU0Fycp8Gm51WDW+RFFX7rDU/7bfImzfpKLXfiRY8bHFhTdndmCHolQUTbx+Q29H9aQJ5YEubLJX3fvcQDMc1bgKludNYHwq+GbLh+FEpi2/91/bqdxoeWWp6elRS/Rezs4sWeedTSQKM9eU8WzAg1w2hdTWXnQFLei1dug991iH/wVA51Dq36p0kYIkZVXfKfTihdrToQwl//536xmc/bAuLKBVF7ERLaAUU8dvAlL1FPWnmLn4wU0Bq/V2+XSndhZjEp6CGlILQFT0PpeRdXt0Hj5PBaKlSaqVrba+hoqBgG+GO6Oi5Q==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kOmI6e+5nI4HIPExrI2uUAqUlOfdfjN76diyyic3nIE=; b=ISyuUoNicJpjtWj4RPcX+Us27yRLi6S6gwxnhLuISD3PN/bz/Wq/s0o0bgqe9wyonZ6A0okg/o7cBeoTMvEgvp4HpiL4xrTmdeoF/WeF8+N/jDisI1tLFWZQEW5vJahdrHj9O6AOove8h63YO6Fpsu11jKeI23VE3qFtVCScTADQPvb0ikdrps3oBfak36Oi3e+a8lpPk0hLe3GabLCMQNF6pRYUjkyWMkD7yS+4eEAo/B1otMbANl7w+N0l/vIw/o2mYllqXdWfDEoaVkJkJJ1BhqAC88sHiKtXAxuR/n6yKjD2377oLB42aCoGdk3aJTale4jqr4NYCFi65al+aA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=stewe.org; dmarc=pass action=none header.from=stewe.org; dkim=pass header.d=stewe.org; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=steweorg.onmicrosoft.com; s=selector2-steweorg-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kOmI6e+5nI4HIPExrI2uUAqUlOfdfjN76diyyic3nIE=; b=Z7ZtQoC/YLiGoPo3QbHPKFww2MyPxMUEDYOt5160IMm2BotswRdNjdBIRAmfEiqa9c0yh/UyrKd9rF2/JgGy3Rs+EhSruzeV1BKVxqZsgvoi6mnfwVYzSsHSDs9a2FIw4ubNo2XapYNTLlr8JtA7WpenOKd3YXqLW52cxI2KrBQ=
Received: from DM6PR17MB3036.namprd17.prod.outlook.com (2603:10b6:5:12e::14) by DM5PR1701MB1802.namprd17.prod.outlook.com (2603:10b6:4:1f::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3676.25; Fri, 18 Dec 2020 21:55:33 +0000
Received: from DM6PR17MB3036.namprd17.prod.outlook.com ([fe80::10eb:387c:5af9:8294]) by DM6PR17MB3036.namprd17.prod.outlook.com ([fe80::10eb:387c:5af9:8294%7]) with mapi id 15.20.3654.025; Fri, 18 Dec 2020 21:55:32 +0000
From: Stephan Wenger <stewe@stewe.org>
To: Alexandre GOUAILLARD <agouaillard@gmail.com>, "Sanchez de la Fuente, Yago" <yago.sanchez@hhi.fraunhofer.de>
CC: "avt@ietf.org" <avt@ietf.org>
Thread-Topic: [AVTCORE] [External] VVC payload format, open issue cluster #1 (SDP for scalability)
Thread-Index: AQHW1TIBY2/VwGx5+kGQZVtX7jlBJqn84IIA
Date: Fri, 18 Dec 2020 21:55:32 +0000
Message-ID: <23B2C082-48E5-4BD7-9AFF-6FC09F164A8E@stewe.org>
References: <8F4FC280-86C4-4570-9745-C5489C12EBD4@stewe.org> <098401d6d49c$ee26b870$ca742950$@bytedance.com> <970DD9A3-E7B6-4F26-9906-5C693B2335D1@hhi.fraunhofer.de> <CAHgZEq7JUDLt+5M3FA5Yv7MTmUrUSVXc8spdEpLmTtkNNqFiXA@mail.gmail.com>
In-Reply-To: <CAHgZEq7JUDLt+5M3FA5Yv7MTmUrUSVXc8spdEpLmTtkNNqFiXA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/16.42.20101102
authentication-results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=stewe.org;
x-originating-ip: [2601:640:8300:1f:281d:32f6:ad19:a4dd]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: b65ca290-9787-456c-2c4f-08d8a39fa497
x-ms-traffictypediagnostic: DM5PR1701MB1802:
x-microsoft-antispam-prvs: <DM5PR1701MB1802E263352F5BBDF7A173ACAEC30@DM5PR1701MB1802.namprd17.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: NUFFOqY1rFbDlhoIq9uEDPS31r+65joKlHGRL6bZqCqKniZN1NR2tT3cimXgWerWjrjHE8qCGSvpylAcKUoiweElF799ns6Td/LrhqGc5lvA5macUf5DTmCCI7AwJ0pHPswB71Q2l956sxFgXVdp5yspizQIccqDdTH+Bb2XFjxf2VyGGlvENDbSOzRQHx8IOKI/ryF5sNB5ysgPzNrDyakIQK+kotTpIwNvdMPO1fePhnz0prKZdj+ViM99FZPnQvYX7juaNL7hDBISTywGnjoyi7dm0Cg5rJkhcsgPTu0bmSSjOrk4uU51UG5aw0I7Y5TEj/2y9UiAmYcalWTCHhj+zvPH6QNlXL/G1FcpVMYY+nst6NtSwRqinM2ShsVChEb3ZFCmDP6DVbcepA+ov6TOQElCqyLBLtcAESRcTOBHHUrlYBBLzO+J4WX6+DeUz4L0iuGHrNeMfvkgp7j3eCEd2Kk1z426b5YEK3GiH80K3+Yt9IupyXRvZFICknBXIqVvJi2fgy+EYcEXUW6DRGJjFNoeJp2cfbOLahRo2G1PkYyzJoe82b9dpb58QI6wLdoWt4/4kN6PYi4Y4tdKoA==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR17MB3036.namprd17.prod.outlook.com; PTR:; CAT:NONE; SFS:(136003)(396003)(366004)(39830400003)(346002)(376002)(316002)(91956017)(2616005)(66556008)(166002)(33656002)(8676002)(76116006)(4326008)(966005)(45080400002)(2906002)(36756003)(86362001)(71200400001)(8936002)(66574015)(64756008)(110136005)(53546011)(5660300002)(6486002)(6506007)(186003)(6512007)(66946007)(508600001)(66446008)(66476007)(83080400002)(83380400001)(17413003)(45980500001); DIR:OUT; SFP:1102;
x-ms-exchange-antispam-messagedata: RurfzHIauB/dgi/p88gEO7eMAtkxOa/gamG/aTyIdG9Az7wd8L6lxt7PUovdth5fv5tDEfQZfD7yMxLoGkJD0xTlXey5Gs63U6RB90T64otI9kK4h2rusddBTRbiCRywRTEWa4HOSgFpr1YkC5yfzivVPKz+3ovNRPuLsHWUZYP9U0veYKxIVcKXpQdEy2WeXZebZKFu9Oz3W3ityF/YuDMFxS++nOtSeO+EuXIWbhSWdD5WZ1GQynux61EvpweYGZhf4LSs37exUyJpwB2IvsFJdlfOxZ0pv6i1JtSgWBr/8s23f3Y0OlE+vFpw3lmlFXaGW0cKj2tYTeDVkuc7a69InKaFudk4pvTIhQpklvO3rCfc7t3ir4DN9tGt7g1dnP0mDOMZ+/5VJYMr586a5f9oR3V9O5odxXq2Y2jjfHiA0jMdzojmS89lxBPJKycJDvuQGvenuPirkZvlzlQ34VZrviJl67fm3HFjlWfyVrMDVglnRj7SyBU4zIOPPnzZxadfOLMYkcVslp3mHZi6KkoburoydTdzsFGBqxVd5zw7ebda4o+mZgk3cxORgVGLcUmOqP7PVjjySxNOTAKBZvGglpznMRyiWiz+n7+1V0g3QcNbMdev7ekjas7Y3of2G/DReD/KohF9eBaUd0SeESR+WkJ3TTUfoEcxX/3tb5Dl1znGCYjU8Sx5HtJW2O9Ne0NSGT1jtU2yXJVcpM/oyXaKdKPRX+ysV9E5U/e5MushpqWDnnsQRn/zEr8h2rHS9UI95hYz2E7DgVHdpTvW3PE8Zqaya9J1KdffyFXJJ1YpOP4x/FxFbZEueXMW/5f9TnmpC/4ijB5JHRfHglkPTcZZprqAweOKb25TL2JUgUwxeUrQzNd0TZ6idZGfB7/aGDMOIBbKm35KRUX7E7jAMdT9EmhlVP9btWN0qREQjs09qAD0rFuDuTwYtiKn1/+dEzhbcv06HoQLuHby4NGTMJb4AtJyWh7LL+YkRi/ngryGp9CqesKS8qRTBSUOTbeZS2BRmxwrKj9WIsLT8n/lrDSD177Gle3OGu/vVCpXL6Zd0n0MKRkvbw9oR6dcp0Zm
x-ms-exchange-transport-forked: True
Content-Type: multipart/alternative; boundary="_000_23B2C08248E54BD79AFF6FC09F164A8Esteweorg_"
MIME-Version: 1.0
X-OriginatorOrg: stewe.org
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DM6PR17MB3036.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b65ca290-9787-456c-2c4f-08d8a39fa497
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Dec 2020 21:55:32.5435 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 865fc51c-5fae-4322-98ef-0121a85df0b6
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: TdivRPwv9m+4eQ+j4nIzpGfnSd8uFrWfeGtDzqQtdQBjHp5BZP+kI2o2T8ueO53K
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1701MB1802
Archived-At: <https://mailarchive.ietf.org/arch/msg/avt/ILDDu4ZFOWLK5CQ6U6hL8zic-bg>
Subject: Re: [AVTCORE] [External] VVC payload format, open issue cluster #1 (SDP for scalability)
X-BeenThere: avt@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Audio/Video Transport Core Maintenance <avt.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/avt>, <mailto:avt-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/avt/>
List-Post: <mailto:avt@ietf.org>
List-Help: <mailto:avt-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/avt>, <mailto:avt-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 18 Dec 2020 21:55:40 -0000

Sure.  How does a week later (13th) sound for the next version?  Some of us will be busy with MPEG, but unless there is a deluge of useful comments during that week, I think we will cope.
No, regarding MMUSIC.  The design of the offer-answer part of an RTP payload spec is AVT territory.  We do not plan to change o/a per se.
S.


From: Alexandre GOUAILLARD <agouaillard@gmail.com>
Date: Friday, December 18, 2020 at 03:36
To: "Sanchez de la Fuente, Yago" <yago.sanchez@hhi.fraunhofer.de>
Cc: Stephan Wenger <stewe@stewe.org>, "avt@ietf.org" <avt@ietf.org>
Subject: Re: [AVTCORE] [External] VVC payload format, open issue cluster #1 (SDP for scalability)

Hi stefan,

Some of the AVTCORE members who have worked on the design of RTP payload for AV1, VP9 and other SVC / Simulcast features in webrtc, included but not limited to SDP O/A, RID and others, are on vacation until january 7th.

I think your proposal would benefit a lot from their spending time and providing feedback from an informed position, and I would humbly request that you push back your deadline for them to participate in the discussion. I do believe that the discussion can go for as long as the chairs want, and there is usually no deadline per say anyway, but it's alway nicer to ask.

Also, I believe that anything related to SDP O/A and its modifications needs to be proposed to the MMUSIC group.

Regards.



On Fri, Dec 18, 2020 at 4:48 PM Sanchez de la Fuente, Yago <yago.sanchez@hhi.fraunhofer.de<mailto:yago.sanchez@hhi.fraunhofer.de>> wrote:
Thanks Stephan,

Your suggestion looks good to me as well.

Best regards,
Yago Sánchez

---
Department Video Coding & Analytics
Group Multimedia Communications

Fraunhofer HHI - Heinrich Hertz Institute
Einsteinufer 37, 10587 Berlin, Germany
http://www.hhi.fraunhofer.de/ip/mc

Tel.: +49 30 310 02663
yago.sanchez@hhi.fraunhofer.de<mailto:yago.sanchez@hhi.fraunhofer.de>




On 17. Dec 2020, at 18:49, Ye-Kui Wang <yekui.wang@bytedance.com<mailto:yekui.wang@bytedance.com>> wrote:

Thanks Stephan for the long email (which is sort of usual for you anyway 😊).

The suggestion looks good to me.

BR, YK

From: avt <avt-bounces@ietf.org<mailto:avt-bounces@ietf.org>> On Behalf Of Stephan Wenger
Sent: Thursday, December 17, 2020 8:31
To: avt@ietf.org<mailto:avt@ietf.org>
Subject: [External] [AVTCORE] VVC payload format, open issue cluster #1 (SDP for scalability)

Hi,
This is the first in a series of emails in which we authors will try to propose ways to solve open issues indicated in the draft (https://tools.ietf.org/html/draft-ietf-avtcore-rtp-vvc-06).  As we have many of those, we will try to organize those open issues into meaningful clusters get piecemeal agreement on each cluster.  Once received, we will update the draft and go for the next cluster.  Sometimes, we will overlap unrelated clusters.

Many of these emails are going to be long, sorry for that.  Many also require quite detailed knowledge of H.266.  Occasionally, we will include some explanatory language on the H.266 feature in question.  If you want to dive deeper, H.266 is now available for free download from https://www.itu.int/itu-t/recommendations/rec.aspx?rec=14336 –have fun reading the 500 pages of super-dense (and small font) text.

The first cluster concerns section 7.1, Media Type Registration, and therein specifically the codepoints and language related to non-temporal scalability support.  This is a hard problem, quite possibly the hardest we need to address in this exercise.  I apologize in advance for the word count.

As a very brief and oversimplified recap, H.266/VVC supports temporal scalability in all of its profiles except the still image profiles.  In that, H.266 is similar to H.265/HEVC.  However, H.266 also has two scalable profiles (one for 4:2:0 and the another for 4:4:4) that allow
for a powerful spatial/SNR scalability mode.  Many of us believe that these profiles have higher chances of success compared to previous video coding standards, because they are a) not requiring much in terms of decoder implementation complexity beyond non-scalable implementations, and b) are conceptually similar to what VP9 and AV1 use, and hence familiar to those using modern scalable codecs, like the webrtc folks.  Insofar, we want to support scalability in the payload format from the outset.

If only temporal scalability is supported, the selection of an appropriate sub-layer in the onion-shaped multi-sublayered stream by an SDP-based offer-answer is a one-dimensional problem, and has been solved in places like the HEVC payload format by an offer-answer exchange involving a single numeric parameter indicative of the highest supported sub-layer.  However, when having all temporal, spatial, and SNR scalability, the optimal selection of an appropriate output layer set (OLS) becomes a complex multi-dimensional problem, that may have more than one solution for different values of “optimal”, and also requires certain context.  We try to solve this multi-dimensional problem by a combination of a requirement for conditional presence of certain otherwise optional SDP codepoints when scalability is in use, and the introduction of an additional few code points.  Alternatives we rejected include not supporting scalability, or relying exclusively on the exchange of possibly many variants of parameter sets, as that could lead to excessively large SDP blobs (easily larger than the 1500 bytes still common as MTU).  Note that the latter option can still be implemented for cases where our restrictions are considered inappropriate.

The design philosophy is as follows:

  1.  When using scalability and OLS-based negotiation, the sender must include a Video Parameter Set (VPS) as a sprop-vps or have otherwise pre-arranged knowledge that the VPS is known by the receiver.  For those unfamiliar with H.266, the VPS includes a “table of content” of the layering structure, and without it, the values of new codepoints (sprop-ols-id and recv-ols-id) could not be interpreted.
  2.  sprop-ols-id will typically point to the “best” output layer set—the combination of layers providing the most pleasing reconstructed result—that the sender can support taking into account all other parameters known at this point of the O/A process.
  3.  The value of sprop-ols-id selects one of possibly several OLSs branches in the VPS.  In the reply, using recv-ols-id, the answerer can select that OLS or other OLSs from the same branch of scalable layers in the VPS to which the sprop-ols-id belongs, but not any other OLSs that may exist in the VPS.  The worst-case fallback will be base-layer only, which is always supported, assuming H.266 and the relevant profiles tiers, and levels are supported.
  4.  Sublayers may be negotiated as in the HEVC payload format and in parallel.
  5.  The interaction between what’s going on for the (temporal) sub-layers and for the (spatial/SNR) layers is an application logic problem and does not need to be specified, nor documented.  It’s very hard to grok for anyone not intimately familiar with H.266.
  6.  With these design principles, sprop-opi is not needed anymore and will be removed.

Based on this design, the following changes can be made in the draft:
-Remove edt note #5, and remove the mentioning of sprop-opi in the sentence above note 5.
-Remove edt note #6, and remove the mentioning of sprop-opi in the sentence above note 6.
-sprop-ols-id: add a sentence “if this optional parameter is present, sprop-vps must also be present or its content must be known a priori at the receiver”
-sprop-recv-id: add “sprop-recv-id, when present, must be included only when sprop-ols-id was received and must refer to an output layer set in the VPS that is in the same dependency tree as the OLS referred to by sprop-ols-id.  If this optional parameter is present, sprop-vps must have been received or its content must be known a priori at the receiver”
-Remove the sprop-opi language and edt note #17

Please provide comments to this design choice by Jan 6th.  Absent comments, we will implement the changes as described above, also any changes required for consistency, and we will add corresponding language in the currently empty offer-answer section of the draft.

Thanks,
Stephan





_______________________________________________
Audio/Video Transport Core Maintenance
avt@ietf.org<mailto:avt@ietf.org>
https://www.ietf.org/mailman/listinfo/avt

_______________________________________________
Audio/Video Transport Core Maintenance
avt@ietf.org<mailto:avt@ietf.org>
https://www.ietf.org/mailman/listinfo/avt


--
Alex. Gouaillard, PhD, PhD, MBA
------------------------------------------------------------------------------------
President - CoSMo Software Consulting, Singapore
------------------------------------------------------------------------------------
sg.linkedin.com/agouaillard<http://sg.linkedin.com/agouaillard>

  *