Re: [rtcweb] Alternative Proposal for Dynamic Codec Parameter Change (Was: Re: Resolution negotiation - a contribution)

Harald Alvestrand <> Wed, 25 April 2012 16:13 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 1AEAF21F87D5 for <>; Wed, 25 Apr 2012 09:13:59 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -110.205
X-Spam-Status: No, score=-110.205 tagged_above=-999 required=5 tests=[AWL=-0.206, BAYES_00=-2.599, J_CHICKENPOX_53=0.6, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id UNOpAYcnFJE7 for <>; Wed, 25 Apr 2012 09:13:58 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id DA05D21F87C9 for <>; Wed, 25 Apr 2012 09:13:57 -0700 (PDT)
Received: from localhost (localhost []) by (Postfix) with ESMTP id BB02639E11A; Wed, 25 Apr 2012 18:13:56 +0200 (CEST)
X-Virus-Scanned: Debian amavisd-new at
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id O-U+qfPsI31n; Wed, 25 Apr 2012 18:13:55 +0200 (CEST)
Received: from ( []) by (Postfix) with ESMTPSA id 0639639E072; Wed, 25 Apr 2012 18:13:54 +0200 (CEST)
Message-ID: <>
Date: Wed, 25 Apr 2012 18:13:54 +0200
From: Harald Alvestrand <>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20120313 Thunderbird/3.1.20
MIME-Version: 1.0
To: Magnus Westerlund <>
References: <> <> <> <>
In-Reply-To: <>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Cc: "" <>
Subject: Re: [rtcweb] Alternative Proposal for Dynamic Codec Parameter Change (Was: Re: Resolution negotiation - a contribution)
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 25 Apr 2012 16:13:59 -0000

On 04/25/2012 04:35 PM, Magnus Westerlund wrote:
> On 2012-04-25 05:20, Justin Uberti wrote:
>> On Tue, Apr 24, 2012 at 10:25 AM, Magnus Westerlund
>> <<>>
>> wrote:
>>      A) Using COP and using SDP based signaling for the dynamic changes are
>>      two quite different models in relation to how the interaction happens.
>>      For COP this all happens in the browser, normally initiated by the
>>      browser's own determination that a COP request or notification is
>>      needed. Harald's proposal appears to require that the JS initiate a
>>      renegotiation. This puts a requirement on the implementation to listen
>>      to the correct callbacks to know when changes happens, such as window
>>      resize. To my knowledge there are not yet any proposals for how the
>>      browser can initiate a JSEP renegotiation.
>> Maybe I misunderstand what you are saying, but the application will
>> definitely know when the browser size changes, and can then trigger a
>> JSEP renegotiation by calling createOffer and shipping it off.
> Yes, they can but that requires them to write code to do this. With COP
> the browser can do it and it works well for all applications, not only
> the ones that have implementors that are aware of the need to do this.
> I am proposing a mechanism that makes the default behavior if you care
> nothing about controlling or optimizing this behavior in your JS to be
> good. In addition JS that wants to control it can affect the behavior
> through generic mechanisms, such as controlling the video element. And
> if clear need is shown and some functionality is missing, W3C can create
> an API to provide explicit control.
It's not at all clear to me that the browser being in control is right 
for all cases.
Renegotiating has a cost, and some of the costs are known only to the 

We may possibly handle this via constraints like 
"VideoEnumRenegotiationTendency" or something like that - when you get 
the proposal together more, I'd like to see the controls you're proposing.
>> I have the opposite concern - how can the browser know when the
>> application makes a change, for instance to display a participant in a
>> large view versus a small view? This may be possible if using a<video/>
>> for display, but I don't think it will be possible when using WebGL for
>> rendering, where the size of the view will be dictated only by the
>> geometry of the scene.
> What I understand from my colleagues is that all rendering of video have
> to go through a video element, even if not displayed directly. Thus the
> application can control the size of the video element used and possibly
> consumed by WebGL to be included in rendering. Thus the application will
> have control over this even when COP is being used.
We have on the "list of things that need doing" the connection of a 
mediastream directly to a canvas, a WebGL element or being captured to a 
file, so don't gamble too much on the video element to stay in the path.

Example code for capturing video to a canvas (taken from a real app):

        video.src = webkitURL.createObjectURL(s);
        canvas.getContext("2d").drawImage(video, 0, 0, 300, 300, 0, 0, 
300, 300);

I am not sure where the size controls go. I'm not even sure the 
developer knows.

> Can you be more specific on any particular usage that you think would
> have issues?
>>      But I am worried that using SDP and with an API that requires the
>>      application to listen for triggers that could benefit from a codec
>>      parameter renegotiation. This will likely only result in good behavior
>>      for the JS application implementors that are really good and find out
>>      what listeners and what signaling tricks are needed with JSEP to get
>>      good performance. I would much rather prefer good behavior by default in
>>      simple applications, i.e. using the default behavior that the browser
>>      implementor have put in.
>> The issue here is that this behavior needs to be sufficiently specified,
>> or else we will have many different behaviors, none of which can be
>> fully overridden by the app. I worry this will make life hard for people
>> trying to develop sophisticated apps.
> I agree that it needs to be clearly specified. But COP appears to be
> less prone to complete failures by using target values. Then at least it
> will receive video that can be used, even if not optimal. imageattr can
> result in failures in the negotiation requiring at minimal a second
> round potentially no agreed on acceptable resolution then that is
> clearly sub-optimal. That is even without considering involving JS
> mainipulating this.
I certainly hope that we can avoid the need for SDP manipulation of JS 
in all of these proposals.