Re: [Sframe] [dispatch] Magnus Westerlund's Block on charter-ietf-sframe-00-00: (with BLOCK and COMMENT)

Sergio Garcia Murillo <> Fri, 11 September 2020 15:32 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 64F2B3A1224; Fri, 11 Sep 2020 08:32:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -3.047
X-Spam-Status: No, score=-3.047 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.948, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id E05Z0A1_maZI; Fri, 11 Sep 2020 08:32:13 -0700 (PDT)
Received: from ( [IPv6:2a00:1450:4864:20::434]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 8D8003A11DF; Fri, 11 Sep 2020 08:32:12 -0700 (PDT)
Received: by with SMTP id c18so11890724wrm.9; Fri, 11 Sep 2020 08:32:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=JuN1ckv9tmyjnAJn5sxuoB9jWH+nbfAB5vXruK38uSk=; b=c0zeEMVvFOysUoXzdUFxSbgdTkwvr2z+LL414tbARkPk9N+M+eTw+2IxRxauxKvu86 qzttmqYEnZaF+ggU/guZPj/tbP5hk3ty+MVuKJy0XG2hFA7lUx6uMCwZkD2tW26Ymz+i YhlMwa3rKX69itdPHS6X6LymdyYT5JLjGH3yXFtky5Osw4QaLyyMEz6c6Crmj1xR/Od5 v8boafywC3eixO/0FR9ziNNXBs9xnH77DzkBF706Xy366WlDkYP/gN3xb9JQWRAkULp3 kZU54WeUCaA0eGs3szTg3GXcqIHLLtjmN07FnufggqnMXC/URkhz+v339dahTSF8kTxc xceQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=JuN1ckv9tmyjnAJn5sxuoB9jWH+nbfAB5vXruK38uSk=; b=tSRGQrNbaKskPxJHjaeHqWpa+o5p/jX6xSukzKijV8RpjTd9N56MasRa1T/+nQWwEx g+Yp4cS7P92S8tZBkuVPRByHhG7pELTFCNzyzTPrDLVQJ+0MfZLkXB2FZNyYHw+LUECY 9pMvJSWOhhl7NDPPd+NUVIFP+L40yVHWZGuGDNidyUZp3oK1Ec3767y8J6MIN7w34kYk simu/HQ6wYqKtfM7ZHOq0JUEum/fi26TlfNoQkVgOydtJokNJ9TBs3qOVlRK1L5h+Fx0 iQD1SjHuJQiu5pfnEMlcVrNWT+UL6hvPbl8nVf+a62p7NxU+bpM3lP9Q6Hj/+Ph8qWjE ZNuQ==
X-Gm-Message-State: AOAM530/JxMeXl9FxLm8b5PyqELfEywwuqrQ8FzXeVH7XewalU5czPdj prVxhexDGS7FbJN/Ttf6tbuPgm/SFwb2bDtZ
X-Google-Smtp-Source: ABdhPJz723uCYagtjNtldVB5mccmc+BdpG2SNiUVbb0rw/CVxmkG5c+nwfKrDDWu3mcf0KoCHkI/og==
X-Received: by 2002:a05:6000:1282:: with SMTP id f2mr2790848wrx.251.1599838329921; Fri, 11 Sep 2020 08:32:09 -0700 (PDT)
Received: from [] ( []) by with ESMTPSA id k6sm5122933wmf.30.2020. (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 11 Sep 2020 08:32:09 -0700 (PDT)
To: Magnus Westerlund <>, "" <>, "" <>, "" <>
Cc: "" <>, "" <>, "" <>, "" <>
References: <> <> <> <> <> <> <>
From: Sergio Garcia Murillo <>
Message-ID: <>
Date: Fri, 11 Sep 2020 17:32:07 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0
MIME-Version: 1.0
In-Reply-To: <>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
Archived-At: <>
Subject: Re: [Sframe] [dispatch] Magnus Westerlund's Block on charter-ietf-sframe-00-00: (with BLOCK and COMMENT)
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 11 Sep 2020 15:32:15 -0000

On 11/09/2020 13:19, Magnus Westerlund wrote:
> Hi,
>>> So do I assume correctly that the idea is that the application layer
>>> using SFRAME if it haves multiple independent or sets of dependent
>>> streams there will be no support for that aspect in SFRAME layer?
>>> Instead it is up to the application to put such information to either
>>> put it inside the encapsualted part of the SFRAME or map it to the
>>> lower transport layer, like to RTP SSRCs and extension information.
>> If i understood you correctly, SFrame is kind of agnostic to the media
>> streams. Currently it has a single global frame counter for all the
> transport, so
>> the number of streams/dependency between them is not known/needed
>> by SFrame.
> Yes, and when using a multi-stream application with layered encoding being
> protected by SFRAME and transported over RTP with repair functions the high
> layer application will need to have a model for how it maps individual
> SFRAMEs to RTP layer functions to enable them to do their work. You get a
> very limited functionality if in an SFU system tries to send all over a
> single RTP SSRC. So this becomes a discussion in the RTP Payload format
> context when carrying SFRAMEs.

I don't really understand your point, let's put an example vp9 svc.

Chrome will encode each picture from the video stream and pass it to the 
vp9 svc encoder. The encoder will produce n-frames (one per spatial 
layer) for a given picture.

Chrome will get each frame with its metadata (spatial/layer id + layer 
structure info + previous frame dependencies + future frame depencies) 
and pass it over to the insertable streams.

The payload will go to javascript, in where SFRAME will encrypt it and 
returned as a binary blob.

The browser will get the binary blob, packetize it in several rtp 
packets according to the new packetization format and add the metadata 
as a header extension.

Is this process what you are describing or is there anything missing?

>>>> However, if the encrypted output is going to be transmitted over RTP
>>>> using a new packetization format, we need to address how to negotiate
>>>> that in the SDP.
>>> Good
>>>> Note that this new packetization format will not only be alid for
>>>> transporting encrypted frames (SFrame or not) but even any other
>>>> audio/video frames.
>>> I understand the potential exists. However, the recommendation for RTP
>>> has been to try to consider Application Level Framing. In the case of
>>> SFRAME that consideration moves up above SFRAME in the choice of how
>>> data is split into SFRAMEs. However, having an RTP paylaod format for
>>> SFRAME, then that should carry SFRAMEs. If you starts putting in other
>>> binary objects into it, then you create a new demultiplexing point for
>>> format between the RTP payload format and the SFRAME processing. That
>> appears unmotivated.
>> Note that SFRAME is not the only encryption possible with w3c insertable
>> streams, it is up to the application to define its own crypto if they
> want. So
>> the payload must be able to transport non-SFRAME opaque blobs.
> So I would consider this a misuse of the RTP payload format. The media type
> will be for SFRAMEs. I understand that you might not be able to prevent this
> in WebRTC. However, from an interoperability point of view you will be
> stating that this is SFRAMEs. RTP will not be able to tell a difference. But
> I don't see that this is will explicitly support carrying other things than
> SFRAME. Because that means bringing in other consideration including the
> security model for the E2E usage.

Not sure if ignoring the reality on how the packetization format is 
going to be used within insertable streams is a good idea.

>>> Should the RTP payload format aspect of the charter be more explicit
>>> in that it needs to disucss the general model of how to use SFRAME and
>>> how an application can use the facilities of RTP to get good
>>> performance from RTP mechanism like FEC and retransmission?
>> I don't think so, RTX and FEC frames MUST not be modified/affected by
>> SFRAME/the new packetization format, so they work out of the box. We can
>> be explicit about that in the chapter as a requirement.
> So I am not saying that RTX or FEC shall be modified. What I am saying is
> that the SFRAME payload format should discuss the impact on the performance
> of these functions depending on how you structure the SFRAMES across SSRCs.
> The most simple example is the one where you put all layers for a video
> encoding in a single SSRC. In such a stream you loose a single RTP packet
> between the transmitting end-point and the SFU. Now the SFU is only
> forwarding the base layer and not the enhancement layers. If base layer and
> enhancement layer share RTP stream, the SFU can't determine if the missing
> packet contains an base layer data or enhancement layer data. If the base
> layer and enhancement layer would be on different SSRC, the fact that there
> is a missing packet on the enhancement layer RTP stream means that the SFU
> can ignore that as it anyway are not forwarding it. This same argument can
> be applied between media sources. So how you map your application level
> structures onto RTP do matter for the resulting performance of RTP. Putting
> all on a single SSRC is not a path to good transport performance.

But that is already the case for SVC codecs, so there is nothing new to 
specify in that regard.

Best regards