Re: [rtcweb] Making progress on the signaling discussion (NB: Action items enclosed!)

[Randell Jesup <[e-mail]>]

On 10/12/2011 9:59 AM, Timothy B. Terriberry wrote:
>> Lets assume we use a subset/variant of SDP as a codec capability
>> description 'language' - (i.e. we won't be using the parts that relate
>> to network properties).
>
> I've seen this proposed a couple of times now, and I would like to point
> out that this is a terrible assumption. There are many things that one
> might want to control about a codec that will never show up in SDP,
> because they don't require explicit negotiation between sender and
> receiver (e.g., the sender gets to make a unilateral decision and the
> receiver simply deals with it... like, say, the amount of time a video
> encoder is willing to spend doing a motion search, etc.).
>
> Limiting yourself to the parameters that do happen to show up in SDP is
> a pretty poor half-solution, if you really do think you need a low-level
> API of this kind, and may not have obvious extension points to get to a
> full solution.

Typically the SDP O-A specifies what you want to receive, and indirectly 
(mostly) what you want to send.  This leaves lots of choices about how 
to achieve that (especially for encoding) up to the application logic at 
either end, once the negotiation is complete.  (As Tim points out.)  The 
"simple" solution would be to embed that application logic in the 
browser code, and feed it the SDP (or equivalent).

We have a more complex case here, where we have the app which may have 
something to say about those choices, or want to vary things during a 
call, etc.  We also have a wider problem space than "video calling"; so 
there may be valid reasons for the app to want different behavior from 
the codecs.

With SDP completely generated by and processed the browser, new video 
codec support is straightforward, which is a good thing.  However, this 
means that the app has more complexity in modifying or controlling the 
negotiation and the way the codecs operate (the "application logic" above).

The problem with providing the app full access to the codec controls and 
negotiation is that it's a severely open-ended problem, and each codec 
is different, with different knobs.  Effectively the app would likely 
support only one or only the original "standard" codecs; existing apps 
wouldn't use (or might break) "new" codecs.

One end of the spectrum effectively gives JS full control of every knob 
in the codec API, which is totally different from one codec to the next. 
  The other end is where you abstract out "common" codec parameters into 
generic values/sliders (image quality vs motion representation, quality 
versus efficiency/frame-rate, etc), which may or may not be handled for 
a given codec (or you punt and give the app no control of these things). 
  Note that the knobs for the same codec (H.264, VP8, Opus, etc) might 
be different in different browsers, because the underlying 
implementation is different.

The downside of the generic approach is that many codec-specific things 
won't be possible to modify, or doing so will be very indirect and might 
have side-effects.

Once again the basic tension between ease-of-use and 
give-me-control-damn-it is a primary factor here.  An added factor here 
is that in the complex model, if a codec is added by the browser, the 
app may not be able to use it (or able to use it above a basic, 
automatic level) without being updated.  You could write an 
codec-interface abstraction JS that isolated the main app from the codec 
complexity (taking the place of the aforementioned sliders, etc), but 
then that has to be updated, which is largely the same problem 
(especially given security concerns).

However, in my mind the more critical issue is ease-of-use (and also 
inter-browser compatibility - two encoders for the same codec may have 
very different knobs at a low level).  Most users of WebRTC won't be 
interested in tweaking the IDR quantization levels; they are more likely 
to be interested in the overall quality/speed/efficiency/frame-rate sort 
of tradeoffs.  Some users may want to tweak each and every parameter and 
buy into updating that code for new codecs (and for new parameters in 
existing codecs).


I'm more than half-tempted to not provide any interface to the codec 
parameters initially, or a very general one (but extensible for future 
use) just covering the most basic tradeoffs only.  Much more than 
half-tempted.

I would in general allow for these parameters to be tweaked during a 
call by the app, though this opens the question about if tweaking them 
can require a re-negotiation of the call.


-- 
Randell Jesup
[e-mail]