Re: [Sframe] Partial decodability and IDUs

Bernard Aboba <> Thu, 19 November 2020 23:13 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 404533A1376 for <>; Thu, 19 Nov 2020 15:13:36 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id SkO0p4MBjzly for <>; Thu, 19 Nov 2020 15:13:34 -0800 (PST)
Received: from ( [IPv6:2a00:1450:4864:20::12f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 27F393A1375 for <>; Thu, 19 Nov 2020 15:13:34 -0800 (PST)
Received: by with SMTP id 74so10755469lfo.5 for <>; Thu, 19 Nov 2020 15:13:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=lEQXersisikeWf7a8vfHCsWPtd8K/ADuREFV9scYyW4=; b=UIxkK4n+jwk6rI7dZk/oxvg07Q3cQme777uNOQIbzUDghbX3sy1gerST4QzoIW1IEf j1zskaCTv1HgwPmYzGs2eGIKs9Y0djk/ffVPZPy6/pCUnAl0H0bez+NR/D7zi03ZeLV9 Vt2dBq5t9SQ/bxF6VI8+XpiXWZM3JsGkFoTzsgDjTfQUUJooRnOv4iUkc8FLavvDdi5t fHajsaYed/O/x6+HIIj33Z54OlKVTqfrB1XaBItkL0HLIZXjMRZnRHReBTdi/wKMxGgz p9stts/AS6LKK0B7DTDbtPRxLemPe6lTlufKECShTtKPvw/SvfROoBk6DHAM7GwiF/wQ SKrg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=lEQXersisikeWf7a8vfHCsWPtd8K/ADuREFV9scYyW4=; b=ENh3OUnlcQczE7B7/wtpGpmyPqIzzhgS+306Njz9uC2sn1B4IGSIGAxe8lyN0TqJDp UZNEkR4eOVz9gFUX51f/XLVoSJUBP1pMo78cJ/egffoL3TcADJGD8EY8FdVBaro/6JYK qBG85fv4vp1LLu8PbBpo2DIM6XXp6xZvSwPn1zhFfO3zyKw/tLp6vIZFp7PpKgVBC1tY 0v1rrzgcKqHX1A3pOrG24PpeNxL+GjgJgi3wBgW5ulCbLNNkonETN50ACgltQsIjw+vx Vd65ZZx0DuvF570TiFiFXKIzG1E/Ms15YIQ1nis7qizHvCqyip0fWZ/+TlkIckYDEJzl +IJA==
X-Gm-Message-State: AOAM533DmOq8wzijHzjNNdif5FLKWzTs0BhRqKSpLyUx1cTp66E31R9v bmcF61patrKiGAFCWBGMpJuwFfX4jFhqZIuIuBb0g859U9oQBg==
X-Google-Smtp-Source: ABdhPJwa+ugSGT4MsNze1c3XkehyP5ZtVTD5ilE+jvXZlUwNqf4mPKyvjhcAZ26IQkSBO6AhPGP5CuA4Y6aCAkB7Jfg=
X-Received: by 2002:a19:88c:: with SMTP id 134mr6309928lfi.560.1605827612103; Thu, 19 Nov 2020 15:13:32 -0800 (PST)
MIME-Version: 1.0
References: <> <> <>
In-Reply-To: <>
From: Bernard Aboba <>
Date: Thu, 19 Nov 2020 15:13:22 -0800
Message-ID: <>
To: Sergio Garcia Murillo <>
Content-Type: multipart/alternative; boundary="00000000000063d9e305b47de0b1"
Archived-At: <>
Subject: Re: [Sframe] Partial decodability and IDUs
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 19 Nov 2020 23:13:36 -0000

Sergio said:

"You are right regarding that the SFrame layer does not need to know what
is feed in for encryption, but in order to be able to have a working end to
end solution for webrtc, someone will need to define what and how this IDUs
are generated and reassembled for each codec if we want to have
interoperable implementations in different devices."

[BA] The job of an SFrame sender is to encrypt and packetize the bitstream
provided by the encoder.  For SFU to act on the packetization in an optimal
way, some rules have to be followed (such as not packetizing frames from
different layers in the same packet). So the sender/packetizer will have
codec-specific logic, so it can parse the bitstream for meta-data, and
figure out how to do the packetization in a manner appropriate for that
codec.  This could include separately packetizing IDUs (slices/tiles), but
I'm not clear this needs to be required for all use cases.

The SFM does not peer into the payload, it only acts on the meta-data
included in RTP header extensions, so the recovery/forwarding/dropping
decision can be largely codec-independent.

The receiver decrypts and de-packetizes the bitstream, then provides it to
the decoder.  The recovery/decryption/de-packetizion process should also be

On Thu, Nov 19, 2020 at 2:11 PM Sergio Garcia Murillo <> wrote:

> You are right regarding that the SFrame layer does not need to know what
> is feed in for encryption, but in order to be able to have a working end to
> end solution for webrtc, someone will need to define what and how this IDUs
> are generated and reassembled for each codec if we want to have
> interoperable implementations in different devices.
> That process is codec-dependant and I would require quite a lot of effort
> (and also supporting it on the agnostic packetization), so I would prefer
> to have strong arguments in favor of doing it.
> On 19/11/2020 22:53, Justin Uberti wrote:
> The encoder needs to be aware of any mechanism to generate IDUs (e.g.,
> slices), and typically each of these IDUs will be handed up to the consumer
> individually. So the SFRAME layer doesn't need to do any splitting, it just
> knows that it should treat each IDU as something it needs to individually
> SFRAME and packetize.
> On Thu, Nov 19, 2020 at 1:40 PM Sergio Garcia Murillo <
>> wrote:
>> Hi all,
>> As most of you already know, this morning I made a presentation in
>> AVTCORE introducing the topic about the need to specify an agnostic video
>> codec packetization format.
>> I got an AP for creating an initial draft so it could be reviewed and
>> accepted.
>> However, there were two main concerns that we should address in this this
>> group:
>>    - Historically, avtcore has explicitly designed not to be payload
>>    agnostic and  declined to standardized codec agnostic payload formats in
>>    number of cases.  If that is to be changed, needs to be done deliberately.
>>    - Need to define the "minimum decoding unit" or "independently
>>    decodable unit", that SFrame will work with.
>> Regarding the second one
>>    - Full video frames (just use whatever is the encoder output)
>>    - Spatial layer frames
>>    - "independend decodable subframes" like h264 slices, vp8 partitions
>>    or av1 tiles which allows partial decodability which is mainly aimed for
>>    enhancing packet loss resilience.
>> Spatial layer frames is the minimum we should target as if not it will
>> just prevent SFUs for using SVC codecs. So the question is if we should go
>> deeper and implement lower partitions of the frames or not.
>> AFAIK, currently, libwertc does not support partial decodability and I
>> personally haven't seen any practical usage of this in the RTC world (while
>> it makes a lot of sense in streaming/broadcasting world), but would like to
>> hear what is the view and experience of the other members of this group.
>> Also note that if we are going to support them on SFrame this will require
>> a greater effort because we will need to explicitly define how the frames
>> must be split before being encrypted y SFrame for *each* possible video
>> codec (h264,h265,vp8,vp9,av1,...).
>> There was also the question about how/if we should support other codec
>> features like DON/interleaved mode for h264, which I also think we should
>> not support mainly because we are not currently using it on webrtc
>> implementations.
>> What do you think?
>> Best regards
>> Sergio
>> --
>> Sframe mailing list
> --
> Sframe mailing list