Re: [rtcweb] Alternative Proposal for Dynamic Codec Parameter Change (Was: Re: Resolution negotiation - a contribution)

[Justin Uberti <[e-mail]>]

On Mon, Apr 30, 2012 at 2:13 AM, Magnus Westerlund <
[e-mail]> wrote:

> On 2012-04-25 18:13, Harald Alvestrand wrote:
> > On 04/25/2012 04:35 PM, Magnus Westerlund wrote:
> >> On 2012-04-25 05:20, Justin Uberti wrote:
> >>> On Tue, Apr 24, 2012 at 10:25 AM, Magnus Westerlund
> >>> <[e-mail]<mailto:[e-mail]
> >>
> >>> wrote:
> >>>
> >>>      A) Using COP and using SDP based signaling for the dynamic
> changes are
> >>>      two quite different models in relation to how the interaction
> happens.
> >>>
> >>>      For COP this all happens in the browser, normally initiated by the
> >>>      browser's own determination that a COP request or notification is
> >>>      needed. Harald's proposal appears to require that the JS initiate
> a
> >>>      renegotiation. This puts a requirement on the implementation to
> listen
> >>>      to the correct callbacks to know when changes happens, such as
> window
> >>>      resize. To my knowledge there are not yet any proposals for how
> the
> >>>      browser can initiate a JSEP renegotiation.
> >>>
> >>>
> >>> Maybe I misunderstand what you are saying, but the application will
> >>> definitely know when the browser size changes, and can then trigger a
> >>> JSEP renegotiation by calling createOffer and shipping it off.
> >> Yes, they can but that requires them to write code to do this. With COP
> >> the browser can do it and it works well for all applications, not only
> >> the ones that have implementors that are aware of the need to do this.
> >>
> >> I am proposing a mechanism that makes the default behavior if you care
> >> nothing about controlling or optimizing this behavior in your JS to be
> >> good. In addition JS that wants to control it can affect the behavior
> >> through generic mechanisms, such as controlling the video element. And
> >> if clear need is shown and some functionality is missing, W3C can create
> >> an API to provide explicit control.
> > It's not at all clear to me that the browser being in control is right
> > for all cases.
> > Renegotiating has a cost, and some of the costs are known only to the
> > application.
> >
> > We may possibly handle this via constraints like
> > "VideoEnumRenegotiationTendency" or something like that - when you get
> > the proposal together more, I'd like to see the controls you're
> proposing.
> >>> I have the opposite concern - how can the browser know when the
> >>> application makes a change, for instance to display a participant in a
> >>> large view versus a small view? This may be possible if using a<video/>
> >>> for display, but I don't think it will be possible when using WebGL for
> >>> rendering, where the size of the view will be dictated only by the
> >>> geometry of the scene.
> >> What I understand from my colleagues is that all rendering of video have
> >> to go through a video element, even if not displayed directly. Thus the
> >> application can control the size of the video element used and possibly
> >> consumed by WebGL to be included in rendering. Thus the application will
> >> have control over this even when COP is being used.
> > We have on the "list of things that need doing" the connection of a
> > mediastream directly to a canvas, a WebGL element or being captured to a
> > file, so don't gamble too much on the video element to stay in the path.
> >
> > Example code for capturing video to a canvas (taken from a real app):
> >
> >         video.src = webkitURL.createObjectURL(s);
> >         canvas.getContext("2d").drawImage(video, 0, 0, 300, 300, 0, 0,
> > 300, 300);
>
> According to my Colleague Per-Erik you can use video.width and
> video.height to set the resolution the video element should have.
>
> see
> http://dev.w3.org/html5/spec/the-video-element.html#the-video-element
>
> Thus the code you presented should be something like.
>
> video.src = webkitURL.createObjectURL(s);
> video.width = 300;
> video.height = 300;
> canvas.getContext("2d").drawImage(video, 0, 0, 300, 300);
>
> That would set the video objects size to a 300, 300 video resolution.
> The drawImage call I have removed the part which selects a cropping of
> the source video to automatically show the full video in the specified
> area.
>
> The adaptation of the video will of course not happen instantly so until
> that happens the video will be scaled to the specified width and height.
> And the actual delivered resolution will depend on the COP request and
> what the sender support.
>

I think this is a reasonable way to approach the problem as long as a
<video/> element stays in the path. But as Harald mentions we have already
considered several usages where a <video/> would not be in the path, i.e.
connecting a MediaStream to a file, or perhaps as input to another
PeerConnection.

Perhaps we need to provide hooks directly on the MediaStream to allow the
resolution to be specified, and these hooks are invoked automatically when
the MediaStream is connected to a <video/>.

>
> >> I agree that it needs to be clearly specified. But COP appears to be
> >> less prone to complete failures by using target values. Then at least it
> >> will receive video that can be used, even if not optimal. imageattr can
> >> result in failures in the negotiation requiring at minimal a second
> >> round potentially no agreed on acceptable resolution then that is
> >> clearly sub-optimal. That is even without considering involving JS
> >> mainipulating this.
> >
> > I certainly hope that we can avoid the need for SDP manipulation of JS
> > in all of these proposals.
> >
>
> Looking forward to a bit more detailed explanation of how that is
> accomplished in relation to the API.
>
> Cheers
>
> Magnus Westerlund
>
> ----------------------------------------------------------------------
> Multimedia Technologies, Ericsson Research EAB/TVM
> ----------------------------------------------------------------------
> Ericsson AB                | Phone  [phone]
> Färögatan 6                | Mobile [phone]
> SE-164 80 Stockholm, Sweden| mailto: [e-mail]
> ----------------------------------------------------------------------
>
>