Re: [rtcweb] Alternative Proposal for Dynamic Codec Parameter Change (Was: Re: Resolution negotiation - a contribution)

Harald Alvestrand <harald@alvestrand.no> Wed, 25 April 2012 16:13 UTC

Return-Path: <harald@alvestrand.no>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1AEAF21F87D5 for <rtcweb@ietfa.amsl.com>; Wed, 25 Apr 2012 09:13:59 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -110.205
X-Spam-Level:
X-Spam-Status: No, score=-110.205 tagged_above=-999 required=5 tests=[AWL=-0.206, BAYES_00=-2.599, J_CHICKENPOX_53=0.6, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UNOpAYcnFJE7 for <rtcweb@ietfa.amsl.com>; Wed, 25 Apr 2012 09:13:58 -0700 (PDT)
Received: from eikenes.alvestrand.no (eikenes.alvestrand.no [158.38.152.233]) by ietfa.amsl.com (Postfix) with ESMTP id DA05D21F87C9 for <rtcweb@ietf.org>; Wed, 25 Apr 2012 09:13:57 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by eikenes.alvestrand.no (Postfix) with ESMTP id BB02639E11A; Wed, 25 Apr 2012 18:13:56 +0200 (CEST)
X-Virus-Scanned: Debian amavisd-new at eikenes.alvestrand.no
Received: from eikenes.alvestrand.no ([127.0.0.1]) by localhost (eikenes.alvestrand.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O-U+qfPsI31n; Wed, 25 Apr 2012 18:13:55 +0200 (CEST)
Received: from hta-dell.lul.corp.google.com (62-20-124-50.customer.telia.com [62.20.124.50]) by eikenes.alvestrand.no (Postfix) with ESMTPSA id 0639639E072; Wed, 25 Apr 2012 18:13:54 +0200 (CEST)
Message-ID: <4F9822C2.7090602@alvestrand.no>
Date: Wed, 25 Apr 2012 18:13:54 +0200
From: Harald Alvestrand <harald@alvestrand.no>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.28) Gecko/20120313 Thunderbird/3.1.20
MIME-Version: 1.0
To: Magnus Westerlund <magnus.westerlund@ericsson.com>
References: <4F869648.2020605@alvestrand.no> <4F96B7C9.1030609@ericsson.com> <CAOJ7v-0ZPxxPEpr6-r8tUyiWJ0MJz47s59kD2tiUPo7Tkj9qcA@mail.gmail.com> <4F980BA6.7070405@ericsson.com>
In-Reply-To: <4F980BA6.7070405@ericsson.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Cc: "rtcweb@ietf.org" <rtcweb@ietf.org>
Subject: Re: [rtcweb] Alternative Proposal for Dynamic Codec Parameter Change (Was: Re: Resolution negotiation - a contribution)
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 25 Apr 2012 16:13:59 -0000

On 04/25/2012 04:35 PM, Magnus Westerlund wrote:
> On 2012-04-25 05:20, Justin Uberti wrote:
>> On Tue, Apr 24, 2012 at 10:25 AM, Magnus Westerlund
>> <magnus.westerlund@ericsson.com<mailto:magnus.westerlund@ericsson.com>>
>> wrote:
>>
>>      A) Using COP and using SDP based signaling for the dynamic changes are
>>      two quite different models in relation to how the interaction happens.
>>
>>      For COP this all happens in the browser, normally initiated by the
>>      browser's own determination that a COP request or notification is
>>      needed. Harald's proposal appears to require that the JS initiate a
>>      renegotiation. This puts a requirement on the implementation to listen
>>      to the correct callbacks to know when changes happens, such as window
>>      resize. To my knowledge there are not yet any proposals for how the
>>      browser can initiate a JSEP renegotiation.
>>
>>
>> Maybe I misunderstand what you are saying, but the application will
>> definitely know when the browser size changes, and can then trigger a
>> JSEP renegotiation by calling createOffer and shipping it off.
> Yes, they can but that requires them to write code to do this. With COP
> the browser can do it and it works well for all applications, not only
> the ones that have implementors that are aware of the need to do this.
>
> I am proposing a mechanism that makes the default behavior if you care
> nothing about controlling or optimizing this behavior in your JS to be
> good. In addition JS that wants to control it can affect the behavior
> through generic mechanisms, such as controlling the video element. And
> if clear need is shown and some functionality is missing, W3C can create
> an API to provide explicit control.
It's not at all clear to me that the browser being in control is right 
for all cases.
Renegotiating has a cost, and some of the costs are known only to the 
application.

We may possibly handle this via constraints like 
"VideoEnumRenegotiationTendency" or something like that - when you get 
the proposal together more, I'd like to see the controls you're proposing.
>> I have the opposite concern - how can the browser know when the
>> application makes a change, for instance to display a participant in a
>> large view versus a small view? This may be possible if using a<video/>
>> for display, but I don't think it will be possible when using WebGL for
>> rendering, where the size of the view will be dictated only by the
>> geometry of the scene.
> What I understand from my colleagues is that all rendering of video have
> to go through a video element, even if not displayed directly. Thus the
> application can control the size of the video element used and possibly
> consumed by WebGL to be included in rendering. Thus the application will
> have control over this even when COP is being used.
We have on the "list of things that need doing" the connection of a 
mediastream directly to a canvas, a WebGL element or being captured to a 
file, so don't gamble too much on the video element to stay in the path.

Example code for capturing video to a canvas (taken from a real app):

        video.src = webkitURL.createObjectURL(s);
        canvas.getContext("2d").drawImage(video, 0, 0, 300, 300, 0, 0, 
300, 300);

I am not sure where the size controls go. I'm not even sure the 
developer knows.

> Can you be more specific on any particular usage that you think would
> have issues?
>
>
>
>>      But I am worried that using SDP and with an API that requires the
>>      application to listen for triggers that could benefit from a codec
>>      parameter renegotiation. This will likely only result in good behavior
>>      for the JS application implementors that are really good and find out
>>      what listeners and what signaling tricks are needed with JSEP to get
>>      good performance. I would much rather prefer good behavior by default in
>>      simple applications, i.e. using the default behavior that the browser
>>      implementor have put in.
>>
>>
>> The issue here is that this behavior needs to be sufficiently specified,
>> or else we will have many different behaviors, none of which can be
>> fully overridden by the app. I worry this will make life hard for people
>> trying to develop sophisticated apps.
>
> I agree that it needs to be clearly specified. But COP appears to be
> less prone to complete failures by using target values. Then at least it
> will receive video that can be used, even if not optimal. imageattr can
> result in failures in the negotiation requiring at minimal a second
> round potentially no agreed on acceptable resolution then that is
> clearly sub-optimal. That is even without considering involving JS
> mainipulating this.
I certainly hope that we can avoid the need for SDP manipulation of JS 
in all of these proposals.