Re: [Sframe] Partial decodability and IDUs

Sergio Garcia Murillo <> Thu, 19 November 2020 22:11 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 10E483A0B18 for <>; Thu, 19 Nov 2020 14:11:23 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id q_plGTGNlPsO for <>; Thu, 19 Nov 2020 14:11:21 -0800 (PST)
Received: from ( [IPv6:2a00:1450:4864:20::42f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id DA0603A0DAB for <>; Thu, 19 Nov 2020 14:11:20 -0800 (PST)
Received: by with SMTP id j7so8066647wrp.3 for <>; Thu, 19 Nov 2020 14:11:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=I05X5lXdPkG6iHplTGlEGBpUW7HxQdHH0NLlBhSnxYI=; b=x6zkupURyGiSgL6Xezd8L0GMJREvt4VjfaYYog2JQ+Td5drPUjpVb0AUzCzItoUTae X+PHV2K9vyXl5AGjw2K6PQGA2OF4vlhNUrtgSVzQdP/XIg2/T7kzlkFMw4HMMbZ7orC7 nCRrs0YKocpzIxDQT0npcoHnWgjskZUtOozOkvODPCV8w8frGSqhb99oO6USpdHuaUgi XHjTr3W2v4Yc6N2HYgKm12LHuM9L/6iAePWe+TcEX5RVWtJWVOdNgj3+jXs8AbvOfYEE A5PBmyXoYBXjxzUySYvd+zjI97yq665VDCp6LXVK8ZjlnFwRs94q4F88mDPVteyes4ey XSrQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=I05X5lXdPkG6iHplTGlEGBpUW7HxQdHH0NLlBhSnxYI=; b=LoxfdVdJ5mkrDQADob0Ql02XHtMAEUUK95rSz6LI9pIlvxSSOdvVFNe9Ra/U1A3qcE xKQ0wvlWZdrfBXfyXKW+tQUNVFumcw6wBb/dhnS0/eFwYgpmgWFAVG5iSRqakrny9V+c rnBSISI94qdP4mPiNqkYzr31JVNHhE9G6vcYE00nLWKpL2rYFf0rO0sdixEYFTtt/K4+ U1Qgo+FNhkpO8mIYHm1KL58aicwpUyXS8qyE6WeVaupw1G1ZTEy8wQTkYQazKtUoQKjS sIv0ez6vV0aykyRgykhDRSA3lcUYExE5i6tvW3vXa1K1djAA3eCKtULfKjszbGXSVpaq Q+Xg==
X-Gm-Message-State: AOAM532AQBsp8RCfFBrX0dFgyDg+8HOo6pCySyE82z8xWBEh4+mu+Gn9 BSPWxukLRs/2HMMYCtTG7iJhrulISj3WMg==
X-Google-Smtp-Source: ABdhPJwmFecvg4x8+hXJpUY6fIJPQpP1UvAzcLbXBfIlz++uD3V3XALZJZvx1GIrD9F6N5yUz1aFRA==
X-Received: by 2002:a5d:4cca:: with SMTP id c10mr13118210wrt.372.1605823878755; Thu, 19 Nov 2020 14:11:18 -0800 (PST)
Received: from [] ( []) by with ESMTPSA id y4sm1916733wmj.2.2020. for <> (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 19 Nov 2020 14:11:18 -0800 (PST)
References: <> <>
From: Sergio Garcia Murillo <>
Message-ID: <>
Date: Thu, 19 Nov 2020 23:11:16 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.4.3
MIME-Version: 1.0
In-Reply-To: <>
Content-Type: multipart/alternative; boundary="------------6C2FD14850A9AA83381C2325"
Content-Language: en-US
Archived-At: <>
Subject: Re: [Sframe] Partial decodability and IDUs
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 19 Nov 2020 22:11:23 -0000

You are right regarding that the SFrame layer does not need to know what 
is feed in for encryption, but in order to be able to have a working end 
to end solution for webrtc, someone will need to define what and how 
this IDUs are generated and reassembled for each codec if we want to 
have interoperable implementations in different devices.

That process is codec-dependant and I would require quite a lot of 
effort (and also supporting it on the agnostic packetization), so I 
would prefer to have strong arguments in favor of doing it.

On 19/11/2020 22:53, Justin Uberti wrote:
> The encoder needs to be aware of any mechanism to generate IDUs (e.g., 
> slices), and typically each of these IDUs will be handed up to the 
> consumer individually. So the SFRAME layer doesn't need to do any 
> splitting, it just knows that it should treat each IDU as something it 
> needs to individually SFRAME and packetize.
> On Thu, Nov 19, 2020 at 1:40 PM Sergio Garcia Murillo 
> < 
> <>> wrote:
>     Hi all,
>     As most of you already know, this morning I made a presentation in
>     AVTCORE introducing the topic about the need to specify an
>     agnostic video codec packetization format.
>     <>
>     I got an AP for creating an initial draft so it could be reviewed
>     and accepted.
>     However, there were two main concerns that we should address in
>     this this group:
>       * Historically, avtcore has explicitly designed not to be
>         payload agnostic and  declined to standardized codec agnostic
>         payload formats in  number of cases.  If that is to be
>         changed, needs to be done deliberately.
>       * Need to define the "minimum decoding unit" or "independently
>         decodable unit", that SFrame will work with.
>     Regarding the second one
>       * Full video frames (just use whatever is the encoder output)
>       * Spatial layer frames
>       * "independend decodable subframes" like h264 slices, vp8
>         partitions or av1 tiles which allows partial decodability
>         which is mainly aimed for enhancing packet loss resilience.
>     Spatial layer frames is the minimum we should target as if not it
>     will just prevent SFUs for using SVC codecs. So the question is if
>     we should go deeper and implement lower partitions of the frames
>     or not.
>     AFAIK, currently, libwertc does not support partial decodability
>     and I personally haven't seen any practical usage of this in the
>     RTC world (while it makes a lot of sense in streaming/broadcasting
>     world), but would like to hear what is the view and experience of
>     the other members of this group. Also note that if we are going to
>     support them on SFrame this will require a greater effort because
>     we will need to explicitly define how the frames must be split
>     before being encrypted y SFrame for *each* possible video codec
>     (h264,h265,vp8,vp9,av1,...).
>     There was also the question about how/if we should support other
>     codec features like DON/interleaved mode for h264, which I also
>     think we should not support mainly because we are not currently
>     using it on webrtc implementations.
>     What do you think?
>     Best regards
>     Sergio
>     -- 
>     Sframe mailing list
> <>
>     <>