[rtcweb] codec and connection negotiation

[Matthew Kaufman <[e-mail]>]

I put this together to try to help myself, and perhaps others, 
understand the various ways in which codec and connection negotiation 
might be decomposed and then put together for RTCWEB applications. It is 
a bit of a stream of consciousness, but hopefully we can get some good 
discussion provoked.

A. How to encode codec capabilities or choices

A1 - Don't provide a canonical way of encoding codec settings. The 
choice of which codec and what parameters is encoded in an ad-hoc way 
that requires no standardization.

A2 - Encode codec settings using SDP. The choice of which codec and what 
parameters is encoded using the syntax and semantics of SDP and reuses 
the SDP standardization work.

A3 - Encode codec settings using a JSON encoding of SDP. The choice of 
which codec and what parameters is using the semantics of SDP but a JSON 
syntax. Reuses the SDP standardization plus new standards around how to 
encode SDP in JSON.

B. How to expose codec capabilities and controls

B1. Expose codec capabilities and settings via individual Javascript 
APIs. For example one might say "camera.encodeMode = "H.264"; 
camera.encodeWidth = 640;" or "if (camera.canEncode("H.264")) ..."

B2. Expose codec capabilities and settings via a combined Javascript API 
that concatenates all the capabilities and settings into a single 
object. Calling "camera.encodeCapabilities()" would return a large 
object that is a full enumeration of what it can do.

B3. Expose codec capabilities and settings via a combined Javascript API 
that produces and consumes SDP strings.

B4. Don't expose codec capbilities and settings via Javascript, rather 
handle this outside of Javascript.

C. How to agree on which codec and settings to use

C1. The server receives information about the capabilities, decides what 
settings should be used, and communicates them to the endpoints

C2. The endpoints agree using offer-answer, using the server as a 
communication channel

C3. The endpoints agree using offer-answer, directly between the two 
endpoints


(There's obviously a few additional nuances, like whether to use 
something like RFC 5939 SDP capability negotiation, that aren't 
well-captured above.)

Now, with the above, we find that some of the models that have been 
discussed are combinations of the above choices.

X1 - Use A2, B3, C2 - The endpoints generate SDP for offer-answer by 
firing an SDP event, having it sent via a server to the far end, where 
it is injected as SDP into the Javascript.

X2 - Use A1, B1, C1 - The endpoints run Javascript that inspects their 
capabilities, sends that to the server which decides what is best, and 
sends back information that the Javascript uses to set up the encoders.

X3 - Use A2, B4, C3 - The endpoints establish a communication channel 
using ICE and an out-of-band channel, they then use SDP inside of SIP to 
directly negotiate using offer-answer, the server has no information 
about what was chosen and does not participate.

We also find that almost all the models can map, though with varying 
degrees of pain, to others.

For instance:

Y1 - Use A2, B1, C2 - The endpoints run Javascript that inspects their 
capabilities and encodes it into an SDP string. This is sent via the 
server to the far end, where the SDP is parsed in Javascript and the 
individual APIs called to implement SDP offer-answer.

Y2 - Use A1, B3, C1 - The endpoints generate SDP for offer-answer by 
firing an SDP event. The endpoint then runs Javascript that extracts 
from the SDP all the individual parameters and re-encodes them using an 
ad-hoc scheme. The server determines what each end should do and sends 
ad-hoc messages to the endpoints which then generate the SDP answer that 
causes the desired outcome.

It should be noted that extracting a full set of capabilities when only 
offer-answer is available via the API is particularly painful, as it 
might be necessary to explore a very wide space of fake offers to get 
answers that clarify things like "can do G.722 audio but only if not 
doing H.264 video".

We might be able to learn from some previous experiences:

1. Web-based email. Web browsers run Javascript that uses ad-hoc 
signaling to/from the web site in order to send and receive email. The 
browser does not have SMTP, POP, or IMAP implementations, or even know 
how to parse RFC822 headers on its own. This argues for A1, B1 (or maybe 
B2), C1.

2. Web-based image display. Web browsers send an "Accept" header via 
HTTP indicating which MIME types they can handle. Knowing whether a 
browser can do JPEG is as simple as looking for "image/jpeg" in the 
header. This is probably at the wrong layer for sophisticated web 
applications (as this isn't about what comes back over HTTP, but what 
can be supported over a separate channel), but might be appropriate if 
we could agree on a small number of supported codecs and simply have the 
browser tell the server in a similar manner whether or not it can do RTCWEB.

3. SIP. SIP endpoints use SDP directly between the endpoints (though 
possibly with intermediaries that rarely change the SDP) to do 
offer-answer. This is most like A2, B3 (or B4), C3 (or C2).

4. MGCP. MGCP uses SDP for both capabilities and settings, but the 
server is in control. This is most like A2, B3, C1.

5. H.323 (H.245) uses a protocol other than SDP for both capabilities 
and settings, but the server is in control. This is like A3 (or some 
other encoding), B2, C1

As for connection negotiation, most of the separation previously also 
applies... again whether to put the ICE candidates into SDP or not, 
whether to have APIs that take blobs or individual calls, etc.

But the other issue that arises when we start talking about connection 
negotiation is that if you choose SDP offer-answer for codec 
negotiation, and SDP for ICE candidates, there is the temptation to 
combine the SDP for the two, thus conflating the process of establishing 
a logical channel over which various types of media might flow, and 
establishing just which media should be sent over that channel.

Matthew Kaufman