Re: [Sframe] Partial decodability and IDUs

Magnus Westerlund <magnus.westerlund@ericsson.com> Fri, 20 November 2020 16:01 UTC

Return-Path: <magnus.westerlund@ericsson.com>
X-Original-To: sframe@ietfa.amsl.com
Delivered-To: sframe@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B55803A0D9B for <sframe@ietfa.amsl.com>; Fri, 20 Nov 2020 08:01:11 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.102
X-Spam-Level:
X-Spam-Status: No, score=-2.102 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=ericsson.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cIwJrJJOhJd9 for <sframe@ietfa.amsl.com>; Fri, 20 Nov 2020 08:01:09 -0800 (PST)
Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-eopbgr150081.outbound.protection.outlook.com [40.107.15.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 66AF23A0D97 for <sframe@ietf.org>; Fri, 20 Nov 2020 08:01:08 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jH9AT4anxo+FokbD+HKDVe4ytcwFdQPc/GTU6BkTziTERyU1xGrxYKXLfUmtlY0cf1wH0T9nz9yQYMBL6UuEQWnj02Ish0/gjuT6BqksroQ14Mu+aotLU8kMlKx99/IL+NUw/cZ4uBaecnb9HNOuq6/c7w0Xlwra4kesbIXazANxOvWsgx0XsuvX0HRIzmBfRjT3hcOkSPb/QJlZY/Bt5sMZesPApWKOAJnVAgnv5RVwnG47O+GSD4X02WqVvRqCJRnf/qyfAESilP/HeJeVDoUeAyyd/ROkOvfANvHLiYUEk6vXom/TkJiBWHaK+59XnWmltiC6uYp+QgX+y+rafg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=n/0ilMe9kgZA+y7ser2IvORd44jsBUdPAFg6TtKI1jw=; b=UgoEYtJjyBd4yhq8e9wSbXIyAJ8s25BXDtUXHtvsHBzz8CXogeM/RBl1b587VaDECCnxi2yoId6ISLe+b9X0TnIqQ/Lk+QXrVPGKn+WaKC/Y68SPvflFxqts+JXhon9X6RtgB5xi/NBVGfmWoa9Lfp2R2Z169wZRZpR+PeXSMcd8V8pYOSMBaQkVsAZcJfA+DZK+vP/lpqBmGglX9v9OfmeLYCuWnojy86SCZr/MH7IDZ7q8hKwU54tfT7jSkTHVHUO051unuzowf2yuV3yhvc2VkO1tEWb9Lnybw64WGhXFiHF7UO4le4knxjI/g4+Sp9DnAI6+S4bG37XJeT9tLA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ericsson.com; dmarc=pass action=none header.from=ericsson.com; dkim=pass header.d=ericsson.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=n/0ilMe9kgZA+y7ser2IvORd44jsBUdPAFg6TtKI1jw=; b=DMxyXSXsM/Gt4CUQMMQi1xTOu/YbDJdyHqzNWzX5Z0JpktpYJsPJWiqYy3koShOX/S7C2cjc0nv5BIg3XGB5pHoAGfJD+vn1DkbNa/TnXReWS0yOAWrpBk2Nly7NqEHS6CN5F4Fv3T+4WIkke0i4xSX3UC+GHPMtV2bHFgefTqw=
Received: from AM0PR0702MB3764.eurprd07.prod.outlook.com (2603:10a6:208:18::12) by AM0PR07MB5986.eurprd07.prod.outlook.com (2603:10a6:208:110::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.9; Fri, 20 Nov 2020 16:01:04 +0000
Received: from AM0PR0702MB3764.eurprd07.prod.outlook.com ([fe80::8d73:a26d:e1bb:8943]) by AM0PR0702MB3764.eurprd07.prod.outlook.com ([fe80::8d73:a26d:e1bb:8943%6]) with mapi id 15.20.3589.016; Fri, 20 Nov 2020 16:01:04 +0000
From: Magnus Westerlund <magnus.westerlund@ericsson.com>
To: "bernard.aboba@gmail.com" <bernard.aboba@gmail.com>, "sergio.garcia.murillo@cosmosoftware.io" <sergio.garcia.murillo@cosmosoftware.io>
CC: "rlb@ipv.sx" <rlb@ipv.sx>, "sframe@ietf.org" <sframe@ietf.org>
Thread-Topic: [Sframe] Partial decodability and IDUs
Thread-Index: AQHWvryjarx1ZMYfcUuCgRv1dn/2e6nP/3yAgAAE1wCAAAwnAIAABckAgAACCgCAAAE7AIABFauA
Date: Fri, 20 Nov 2020 16:01:04 +0000
Message-ID: <baa4046e0e1892a6122a208c1679a210e0e1579e.camel@ericsson.com>
References: <e9c91b96-b956-5aa9-b78c-9b9414616c77@gmail.com> <CAOJ7v-3U+dS=V3199yduo6192OF=iXe=et6EYF-ZVFv3drR4kw@mail.gmail.com> <7eadfc25-d326-3a02-f88f-bfdc02b9a42c@cosmosoftware.io> <CAL02cgR4qmAuHzD9mRtOXkTL9+LhN=zNOCynapBcrF=c_e4ZmQ@mail.gmail.com> <2752b6f0-ec61-6a19-e17c-0347c451a17c@cosmosoftware.io> <CAOW+2dsSzZBXo3oyKOfkhXQV7LXfymam9Dvu_bzQWJMsn3mcag@mail.gmail.com> <f659bce7-5031-e3c2-b7a7-94756c449c71@cosmosoftware.io>
In-Reply-To: <f659bce7-5031-e3c2-b7a7-94756c449c71@cosmosoftware.io>
Accept-Language: sv-SE, en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
x-mailer: Evolution 3.28.5-0ubuntu0.18.04.2
authentication-results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=ericsson.com;
x-originating-ip: [192.176.1.82]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 3aab7163-9fa3-420b-d40a-08d88d6d7c46
x-ms-traffictypediagnostic: AM0PR07MB5986:
x-microsoft-antispam-prvs: <AM0PR07MB5986A4F82AEC3852EE8525F295FF0@AM0PR07MB5986.eurprd07.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:5236;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: s1u3sL/GWTUlRmLd2A9aT8WlU9I3uh6OMehNa79b+MWeGc8c9fUqzWsDm4BwqfUKjWwgfj2IHw1Pwouv6Qh+0kUokZIue66feFCU5EmC0deQkc8L7eNOSJTPEZvOtzt5JAiBKNRENuVuLw0Tvbf2NPvXD0rqAfp+np9WLMVJaeiM5IfS4RNZRKdpO3UzusANGWhOtuUMM7EAZewim1OV4nRwgUHx+TfXwZw6K4CIPBLH5nzAyTWXb5ZGyxMkG2/A78ihTJWR7asmNdj7K4m//lLXcEub1uAAkmq1DGQE1OIFZD87K3c4nFmwIMRWHj8m2VprK7wMzl8zrVoI9Rlw/mqirqrSLlZZ+tu1RI7IQ8KTjN5Ul0AaiuJV9GyXVk/g
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR0702MB3764.eurprd07.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39860400002)(366004)(376002)(396003)(136003)(346002)(66616009)(86362001)(316002)(99936003)(2616005)(44832011)(71200400001)(2906002)(8936002)(66946007)(6486002)(76116006)(91956017)(66446008)(64756008)(66556008)(66476007)(186003)(83380400001)(36756003)(54906003)(478600001)(8676002)(26005)(110136005)(6512007)(4326008)(5660300002)(6506007)(99106002); DIR:OUT; SFP:1101;
x-ms-exchange-antispam-messagedata: =?utf-8?B?Rlg2aGFHSERzeWFWaE1RaWcrcktmZTU5WXlRNnB4WE1KczRMNUgxRXA3Z1VS?= =?utf-8?B?K1hXYkFwbmFabDc4ZTA4eXFINmF5dnBITHBpYmlTaVNRY3JlTWJlSU1BeU5x?= =?utf-8?B?MUh0eGxWbVZIa2JvNThxR1ZwaFRzRFdoeUZhRzVnRjBHTkVXdU03REJGQ1pa?= =?utf-8?B?aGFHcGMvUnU5Wjgzb2hOZHI2bWFvZVFYcEo5c2FRMUs5c3VDc1p6U0M2cmNS?= =?utf-8?B?SEtyQ1BRWGliRjE2TGozRm13UzdSSjZ6TnBiK2MzZExobVVvWVNBNllMeTNL?= =?utf-8?B?UGFDRTdDWUFTUG5LaG1WZjNzY0ZQd2tteWp0bGlPWllJbU4xVEdIa2hNSU1B?= =?utf-8?B?OUxMZENzNTd6aFNYVEMyRHEzeDg4R2tTZGZDRUZrNUl0NEhHdWhHd0ZjVUdP?= =?utf-8?B?Q1l0aUlUTFFWYXNwMHN4Rk1iSlAxVTNrSWNjY0xrZFRjNE5sMnoxMlBvUmJw?= =?utf-8?B?c3lmMmZMcmVwT2pNMG9YMUxDNEVCOXBzdk03bDBNNmhlMXExLy9TSU5LWDZ0?= =?utf-8?B?MGtJSUxYV0svTGtrQ1FCblBtQzhhY2NaSWlWc0dBMGpjbHpoM2J4U3BLUzlE?= =?utf-8?B?NDI3QjBnS2xHbWhlOVRXM0JBOHJzOFpFNmZXVkhJYVdES3pPbjFERzR4N2Rs?= =?utf-8?B?VURoV1N4QUVKNHpiaEVnZGNLVVVVK3VJeDVFMVVRcyt1UG9zZW56c0s5dW9D?= =?utf-8?B?QWlLUFVhQytYZmZvTlpoY21VNGJQdm9IcnhKRkMrZEl6UjJ3Y3U0TUFIalVs?= =?utf-8?B?aWFRZEc4NW54NDd4amV1bUlvVU9xZzg5Z2x1dTFKNnVHNUw4UncwWWl3L1Mx?= =?utf-8?B?SWNKN0dmSE1rWGM1eGFjWi8yQ3FHbndTeXRBeUtNSFpEV0xwS1h0VzBPeFgw?= =?utf-8?B?MzJNbEJhdzF0Tm1KYlhJclZZZFk4UXZrL0QyVWp5Z0JQM1ExM0tCQ1dRRm5k?= =?utf-8?B?c084U0FwOXJUQTZxcGdnZzROQ2s2RDZzYStZaFVpWGcxb0NlaDcvLzJiWitD?= =?utf-8?B?RS96UVlyRnlvWU0rZldBZFJHZTFOVVlyaXc1OGVObExKcFpnclhFeCt6dWxp?= =?utf-8?B?bEVHQUx5MktCSEliYW9DMlppWnhTQzdnSE9lemJpT1FDR21yeTNkQ0xIM1h1?= =?utf-8?B?WWN1Q1VjTzl5WE9IWUxwS0NyZ0E3NS9QaFRWRmQxMVBueFNoR0pBazVqbDlY?= =?utf-8?B?eFNFVVNROGtVWXJRQ2tySnJnSDZPSm1QUkNPTzM3KzhraXpFdWx0L2hLRElx?= =?utf-8?B?cSs3UXdhS2tnNmk5WEJYTGpLdmduZmV6bzJjSUZzMFF2OWpUSm11SlQ5bjZO?= =?utf-8?Q?lJg3RV+VIqvmT6yDFbzLofKHr7mSYy0/IQ?=
x-ms-exchange-transport-forked: True
Content-Type: multipart/signed; micalg="sha-256"; protocol="application/x-pkcs7-signature"; boundary="=-bfgKan+LDXfZqRQSXF7f"
MIME-Version: 1.0
X-OriginatorOrg: ericsson.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: AM0PR0702MB3764.eurprd07.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3aab7163-9fa3-420b-d40a-08d88d6d7c46
X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Nov 2020 16:01:04.8373 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: f1j5Nzp0OPbS2TQ9u9WBOpFfiQY2JLSjJe1i6OS8vZG8LUiTHsMRJnYZsvwRbuMiloNJY6Jj91ycYJXd2c7RxK9SXbHQ8tLg1JR3t2EBLbA=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR07MB5986
Archived-At: <https://mailarchive.ietf.org/arch/msg/sframe/YjECtlbTu-zksRSHBhFtN5TJGbI>
Subject: Re: [Sframe] Partial decodability and IDUs
X-BeenThere: sframe@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <sframe.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/sframe>, <mailto:sframe-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/sframe/>
List-Post: <mailto:sframe@ietf.org>
List-Help: <mailto:sframe-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/sframe>, <mailto:sframe-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 20 Nov 2020 16:01:12 -0000

Hi,

If one look at this from the RTP payload level what becomes of potential use and
in some case necessary I think are the following.

The sender will put one or more IDU into independent SFRAMEs and provide the
meta data to it that can include the following:
 - Media Stream + Encoder instance
 - Scalability layer and depdendency information
 - Decoding order
 - Associated Capture/Presentation time
 - timestamp for any consumption related time line (if not redundant)
 - If it represent a switch point (IDR, Intra etc)
 - Will produce output (Video marker bit type, i.e. this IDU(s) will result in
decoded output) (likely for which layers)
 - Reliability level (How necessary the data is, parameter sets is necessary for
any decoding, versus a non-referenced slice). 

The one or more IDUs is a trade-off if between overhead and what benefits there
are to encoded different IDUs in indepdendent SFRAMES. For example encoding
H.26x parameterset NALUs independently depend on how one like to use them, or if
they should be teamed with their related "slice" NALUS, in a STAP style
aggregation of NALUs inside the SFRAME. 

I think the point is to look at this what would be needed to enable the
functionality in the RTP payload format and for the SFUs to be able to correctly
select what SFRAMEs to forward to a particular receiver. 

If some implementation makes is easy for themselves and aggregate more in one
SFRAME that should be possible. It is a trade-off to a large degree between
functionality, cost of doing repair of missing parts, and complexity to
implement. 

There are some basic level which should be very easy to accomplish and never
should be gone below becauase then one is truly throwing away the basic
functionalities of RTP. 

I still think one needs to write a bit of analysis of the different media
encoders functionality and what are shared concepts, what have special quirks
that needs to be thought about. We also I think need to find the terminolgy for
what the different aspects like I listed above really should be called and how
they are mapped to different encoders, in audio, video, tactile, etc. 

I will note that we didn't make much headway in the RTP streams debate before we
actually created the taxonomy of that. People where talking past each others. 

I will also note that a very interesting aspect of the reliability puzzle for
these codecs is what you can determine about a missing RTP packet based on that
you know which SSRC and sequence number that is missing. So if you have a
scalable codec and transmit all the packets on a single SSRC, then a missing
sequence number (detected through the gap) will not tell the receiver if it
needs the packet urgently or not (such a difference between a base layer or the
highest enchancement layer). In some applications that might not matter in
otheres it may. So in the later cases you might want to distribute the layers on
different SSRCs so that one can immediately determine how important the
information is. I think that possibility will exist if this RTP payload format
is done correctly.

Cheers

Magnus Westerlund