[Sframe] Partial decodability and IDUs

Sergio Garcia Murillo <sergio.garcia.murillo@gmail.com> Thu, 19 November 2020 21:40 UTC

Return-Path: <sergio.garcia.murillo@gmail.com>
X-Original-To: sframe@ietfa.amsl.com
Delivered-To: sframe@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 102373A1296 for <sframe@ietfa.amsl.com>; Thu, 19 Nov 2020 13:40:34 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level:
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GLZK5xNkNP91 for <sframe@ietfa.amsl.com>; Thu, 19 Nov 2020 13:40:32 -0800 (PST)
Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B69583A1297 for <sframe@ietf.org>; Thu, 19 Nov 2020 13:40:00 -0800 (PST)
Received: by mail-wm1-x32b.google.com with SMTP id p22so8536838wmg.3 for <sframe@ietf.org>; Thu, 19 Nov 2020 13:40:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:subject:message-id:date:user-agent:mime-version :content-language; bh=qaCVQds58ter7U1M3kDtKOwWpN3ByRL3nsBoOVXPyPk=; b=SgSfxWnGusttR7cqNQ7G4dL0mGY19il9THTRrAsDs5CrTfcuvA/fsvmyVO9fezKGBW M5iSBbxmJIWr/LaILRC+1ubYiO8i4sigEvkxpDAwihTq4NE94dHLVS5SfT82CRZaGtvW WAUEx7dLqWzd1fpaxTpdr2QnAoJNYIHQD9sx4iAIgOuFo2s4+GIpXHGiuX5MqJQs26Vr On+oJvJwkOZVPq68NE/qjKfXc9UgJcFj6RpCdiMRJyExuSjf4GumJiZpsN5+YUFQKOZ7 29xfwXMkk4VSmt1W0N5PEmpNDRb7WxJPfoLYFO7lnZvpxXmFR/I/GHh+RwPNeAokGo4d dCBg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-language; bh=qaCVQds58ter7U1M3kDtKOwWpN3ByRL3nsBoOVXPyPk=; b=H5+bxM4B0aI454liAXHYqQfJQybSFRVO2c5G/fWccM70djXOHTUcypb+LRSnGJ98xi /7yaHbioLyWHIXIIAJgc3yk46zr2Dc7SIx7+IANzECX+fPMAx1qxIqs6cxAl/6k426NK avlhF2sNr+6nP5wNXL3cAyIpwRBtw7cjAjRaMSO59dRMDqBC1jgrMDaIefUB6q2Pt3Qb zmwuJsXcnbb4HRIKOv66xOJ08IrO0UG47Q+ysjGB6l9Yh727ngrxlfWIHDD8Zql1G2ut P0iRM5fJ+8mAYFwxk6cFN7/+gwPii8WExUrz2muegW9w3KUfV9+dO1J32uS7X1LUmR5t gPdA==
X-Gm-Message-State: AOAM531T00mfHYAEuVwf1Wrd+Bvtp27/rVqwnL6EB2pzG5BFPgKc3XHV HsfW1HxeTm1HV0aJSXG4tuqjhX5XI9UUtw==
X-Google-Smtp-Source: ABdhPJxI45+4OaptL6x71/FOTe0Zh12lPDbd4Z/8i7gM1vtykcC7KDuzRrjyvlILfzSuq46LNLfQLA==
X-Received: by 2002:a1c:6002:: with SMTP id u2mr6107233wmb.29.1605821998822; Thu, 19 Nov 2020 13:39:58 -0800 (PST)
Received: from [192.168.0.11] (79.108.125.160.dyn.user.ono.com. [79.108.125.160]) by smtp.googlemail.com with ESMTPSA id u16sm1762078wrn.55.2020.11.19.13.39.58 for <sframe@ietf.org> (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 19 Nov 2020 13:39:58 -0800 (PST)
To: "sframe@ietf.org" <sframe@ietf.org>
From: Sergio Garcia Murillo <sergio.garcia.murillo@gmail.com>
Message-ID: <e9c91b96-b956-5aa9-b78c-9b9414616c77@gmail.com>
Date: Thu, 19 Nov 2020 22:39:57 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.4.3
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="------------01A91898AB6973D182B02591"
Content-Language: en-US
Archived-At: <https://mailarchive.ietf.org/arch/msg/sframe/AI--pP44jLlGDG887raH4zXO4m4>
Subject: [Sframe] Partial decodability and IDUs
X-BeenThere: sframe@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <sframe.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/sframe>, <mailto:sframe-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/sframe/>
List-Post: <mailto:sframe@ietf.org>
List-Help: <mailto:sframe-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/sframe>, <mailto:sframe-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 19 Nov 2020 21:40:34 -0000

Hi all,


As most of you already know, this morning I made a presentation in 
AVTCORE introducing the topic about the need to specify an agnostic 
video codec packetization format.

https://datatracker.ietf.org/meeting/109/materials/slides-109-avtcore-sframe-rtp-encapsulation-00


I got an AP for creating an initial draft so it could be reviewed and 
accepted.

However, there were two main concerns that we should address in this 
this group:

  * Historically, avtcore has explicitly designed not to be payload
    agnostic and  declined to standardized codec agnostic payload
    formats in  number of cases.  If that is to be changed, needs to be
    done deliberately.
  * Need to define the "minimum decoding unit" or "independently
    decodable unit", that SFrame will work with.


Regarding the second one

  * Full video frames (just use whatever is the encoder output)
  * Spatial layer frames
  * "independend decodable subframes" like h264 slices, vp8 partitions
    or av1 tiles which allows partial decodability which is mainly aimed
    for enhancing packet loss resilience.


Spatial layer frames is the minimum we should target as if not it will 
just prevent SFUs for using SVC codecs. So the question is if we should 
go deeper and implement lower partitions of the frames or not.


AFAIK, currently, libwertc does not support partial decodability and I 
personally haven't seen any practical usage of this in the RTC world 
(while it makes a lot of sense in streaming/broadcasting world), but 
would like to hear what is the view and experience of the other members 
of this group. Also note that if we are going to support them on SFrame 
this will require a greater effort because we will need to explicitly 
define how the frames must be split before being encrypted y SFrame for 
*each* possible video codec (h264,h265,vp8,vp9,av1,...).


There was also the question about how/if we should support other codec 
features like DON/interleaved mode for h264, which I also think we 
should not support mainly because we are not currently using it on 
webrtc implementations.


What do you think?


Best regards

Sergio