Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)

[Robin Raymond <[e-mail]>]

Seems that this SDP issue keeps coming back with a bit of "I told you 
so" attached. It's truly awful and I've expressed my personal disdain 
previously. I've only kept quiet because I have no desire to stop WebRTC 
despite how personally horrible I find SDP. However bad it is I do want 
WebRTC released and hopefully this existing SDP based API can be 
deprecated -- fast. I want to see Microsoft onboard with WebRTC but I 
fully understand their position. I find it equally horrible for our Open 
Peer P2P signalling although we can "work around" it temporarily but it 
does hinder our ability to innovate. And when I refer to SDP I really 
mean SDP + offer/answer.

As I knew going in this effort is being primarily spearheaded by the SIP 
world and the push for SDP comes from that realm. At first I thought the 
SIP vendors would love it but I've refined my thinking and I think it's 
equally bad for SIP guys too if they were to think about the 
consequences. They are demanding features like crazy in the browser but 
have made the browser the bottle neck for innovation rather than 
allowing all their features to be implemented within JavaScript with 
media engine hooks inside the browser. The SDP produced by browsers 
won't likely be compatible with them anyway and they are likely to 
require SBCs to "fix" it. This is just bad, especially knowing how SIP 
guys love tweaking SIP for their own favourite flavours.

As much as I hate to say it, I think we need to hold off on releasing 
WebRTC as it is until we have a lower level API. SDP offer/answer is not 
going to cut it for the initial release, this first version of the 
standard needs to live on for at least a few years. We must deprecate 
the current WebRTC API in favour of a more suitable low-level 
replacement API. The revision should focus instead on the extensions 
that can be added in the JavaScript layer and only put the necessary 
hooks in the browser at the most basic level for a media and RTC engine 
to be controlled. Let's not hamper innovation!

If we need compatibility with the current WebRTC API it would be easy to 
create a JavaScript shim that supports the current API and allows a more 
long innovative approach. If there is interest in creating such a shim, 
I would be more than happy to be part of that development effort.

I've written more on the subject here but I don't want to post a long 
rant here: 
http://blog.webrtc.is/2013/06/17/sdp-inside-webrtc-is-bad-for-sip/

-Robin

> Martin Thomson <mailto:[e-mail]>
> 17 June, 2013 5:56 PM
> On 17 June 2013 12:26, Peter Thatcher<[e-mail]>  wrote:
>> Yes, I was expecting you to be more supportive.  I'm surprised out how your
>> want "all or nothing".  I'm afraid if our options for a clean API are all or
>> nothing, we'll just end up with nothing.  I'd prefer to try incremental
>> improvements to word toward (eventually) a clean API.
>
> Yes, I apologize if the language seemed strong, but I stand by those words.
>
> The problem that I see with this, and it is a problem with any
> incremental approach, is that it produces two very different
> interaction models for things that are approximately the same
> fundamental operation.
>
> That is, when I add the first video track to a session I perform one
> set of actions.  Then, when I add subsequent tracks, it's different.
>
> This also has all the purported drawbacks of comment 22 with respect
> to usability, whatever they are.  There must have been some because I
> got a lot of heat about that when we discussed it.
>
>> Do you think it is impossible to work toward a clean API in an incremental
>> approach?  If you think it's possible, I'd like to hear your thoughts on
>> how.
>
> The fundamental problem in WebRTC is the Offer/Answer semantics in the
> API.  That's hard to remove now.  I can't see an incremental change
> that would remove that, and that's every bit as much of a problem as
> having to deal with SDP.  That's how we got to comment 22.
>
> I know that you wanted to do some sort of object representation of
> SDP.  That can be done incrementally by adding to
> RTCSessionDescription, or by replacing it entirely, but it doesn't
> really go to the core of the problem.  If you were willing to tolerate
> the fact that there would be two code paths, two control surfaces that
> effectively did the same things.
>
>> By the way, these API additions would greatly minimize the amount of SDP
>> editing necessary for JS clients that don't use SDP for signalling.  And
>> later incremental improvements could reduce it further.  Also, it's no
>> longer necessary to do offer/answer for adding tracks.  It's only the intial
>> PeerConnection setup that needs to do Offer/Answer.  So, it doesn't inherit
>> all the problems quite as much as you described.  It may be slightly
>> abominable, but I certainly consider it less abominable than the SDP editing
>> necessary without it.
>
> If I am forced to do SDP, I'd rather not have something else as well
> unless that something else is getting me something concrete.  What you
> are proposing does too little to advance a cleaner API model to
> justify the extra overhead that it introduces.  It doesn't tackle the
> hard problems.
>
> I appreciate the philosophy behind no-plan, which is not a non-plan in
> any sense.  It addresses a concern that we (unfortunately) didn't
> really touch on in the MMUSIC call this morning.  However, the
> requirements of the-not-no-plan could be far more elegantly addressed
> within the constraints of the current API without adding all those
> extra description bits.
>
> --Martin
> _______________________________________________
> rtcweb mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/rtcweb
> Peter Thatcher <mailto:[e-mail]>
> 17 June, 2013 3:26 PM
> Yes, I was expecting you to be more supportive.  I'm surprised out how 
> your want "all or nothing".  I'm afraid if our options for a clean API 
> are all or nothing, we'll just end up with nothing.  I'd prefer to try 
> incremental improvements to word toward (eventually) a clean API.
>
> Do you think it is impossible to work toward a clean API in an 
> incremental approach?  If you think it's possible, I'd like to hear 
> your thoughts on how.
>
>
> By the way, these API additions would greatly minimize the amount of 
> SDP editing necessary for JS clients that don't use SDP for 
> signalling.  And later incremental improvements could reduce it 
> further.  Also, it's no longer necessary to do offer/answer for adding 
> tracks.  It's only the intial PeerConnection setup that needs to do 
> Offer/Answer.  So, it doesn't inherit all the problems quite as much 
> as you described.  It may be slightly abominable, but I certainly 
> consider it less abominable than the SDP editing necessary without it.
>
>
>
> _______________________________________________
> rtcweb mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/rtcweb
> Martin Thomson <mailto:[e-mail]>
> 17 June, 2013 2:57 PM
> Maybe you'd expect me to be more supportive of something that looked
> so much like CU-RTC-Web. It inherits all the worst properties of JSEP
> (Offer/Answer, SDP editing) with a partial implementation of a clean
> API.
>
> It's comment 22-lite. It's an abomination. If you are going to do
> this, do it properly.
> _______________________________________________
> rtcweb mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/rtcweb
> Peter Thatcher <mailto:[e-mail]>
> 17 June, 2013 8:57 AM
> Google is in full support of "Plan B" for encoding multiple media 
> sources in SDP, and would like to see the Plan A vs. Plan B decision 
> resolved soon.  Recently, though, a third option, called "NoPlan" has 
> been proposed, but it lacked the details of what a JS API would look 
> like for NoPlan.  Cullen asked to see such an API proposal, and so I 
> have worked with Emil to make one.  He will be adding it to the NoPlan 
> draft, but I will also include it in this email.
>
> Again, Google is in full support of "Plan B".  But if Plan A vs. Plan 
> B cannot be decided, then we support NoPlan with the following 
> additions to the WebRTC JS API as an option that allows implementing 
> either Plan A or Plan B  in Javascript.  And even if Plan A vs. Plan B 
> is resolved, these API additions would still be a big improvement for 
> those WebRTC applications that don't use SDP for signalling.
>
> It is a bit long because I have added many comments and examples, but 
> the actually additions only include two new methods on PeerConnection 
> and a few new dictionaries.  So don't be overwhelmed :).
>
>
>
> Intro: This follows the model of createDataChannel, which has a JS 
> method on PeerConnection that makes it possible to add data channels 
> without going through SDP.  Furthermore, just like createDataChannel 
> allows 2 ways to handle neogitation (the "I know what I'm doing; 
>  Here's what I want to send; Let me signal everything" mode and the 
> "please take care of it for me;  send an OPEN message" mode), this 
> also has 2 ways to handle negotiation (the "I know what I'm doing; 
> Here's what I want to send; Let me signal everything" mode and the 
> "please take care of it for me;  send SDP back and forth" mode).
>
> Following the success of createDataChannel, this allows simple 
> applications to Just Work and more advanced applications to easily 
> control what they need to.  In particular, it's possible to use this 
> API to implement either Plan A or Plan B.
>
> // The following two method are added to RTCPeerConnection
> partial interface RTCPeerConnection {
>  // Create a stream that is used to send a source stream.
>  // The MediaSendStream.description can be used for signalling.
>  // No media is sent until addStream(MediaSendStream) is called.
>  LocalMediaStream createLocalStream(MediaStream sourceStream);
>
>  // Create a stream that is used to receive media from the remote side,
>  // given the parameters signalled from MedaiSendStream.description.
>  MediaStream createRemoteStream(MediaStreamDescription description);
> }
>
>
> interface LocalMediaStream implements MediaStream {
>   // This can be changed at any time, but especially before calling
>   // PeerConnection.addStream
>   attribute MediaStreamDescription description;
> }
>
>
> // Represents the parameters used to either send or receive a stream
> // over a PeerConnection.
> dictionary MediaStreamDescription {
>   MediaStreamTrackDescription[] tracks;
> }
>
>
> // Represents the parameters used to either send or receive a track 
> over // a PeerConnection.  A track has many "flows", which can be grouped
> // together.
> dictionary MediaStreamTrackDescription {
>   // Same as the MediaStreamTrack.id
>   DOMString id;
>
>   // Same as the MediaStreamTrack.kind
>   DOMString kind;
>
>   // A track can have many "flows", such as for Simulcast, FEC, etc.
>   // And they can be grouped in arbitrary ways.
>   MediaFlowDescription[] flows;
>   MediaFlowGroup[] flowGroups;
> }
>
> // Represents the parameters used to either send or receive a "flow"
> // over a PeerConnection.  A "flow" is a media that arrives with a
> // single, unique SSRC.  One to many flows together make up the media
> // for a track.  For example, there may be Simulcast, FEC, and RTX
> // flows.
> dictionay MediaFlowDescription {
>   // The "flow id" must be unique to the track, but need not be unique
>   // outside of the track (two tracks could both have a flow with the
>   // same flow ID).
>   DOMString id;
>
>   // Each flow can go over its own transport.  If the JS sets this to a
>   // transportId that doesn't have a transport setup already, the
>   // browser will use SDP negotiation to setup a transport to back that
>   // transportId.  If This is set to an MID in the SDP, then that MID's
>   // transport is used.
>   DOMString transportId;
>
>   // The SSRC used to send the flow.
>   unsigned int ssrc;
>
>   // When used as receive parameters, this indicates the possible list
>   // of codecs that might come in for this flow.  For exmample, a given
>   // receive flow could be setup to receive any of OPUS, ISAC, or PCMU.
>   // When used as send parameters, this indicates that the first codec
>   // should be used, but the browser can use send other codecs if it
>   // needs to because of either bandwidth or CPU constraints.
>   MediaCodecDescription[] codecs;
> }
>
>
> dictionary MediaFlowGroup {
>   DOMString type;  // "SIM" for Simulcast, "FEC" for FEC, etc
>   DOMString[] flowids;
> }
>
> dictionary MediaCodecDescription {
>   unsigned byte payloadType;
>   DOMString name;
>   unsigned int? clockRate;
>   unsigned int? bitRate;
>   // A grab bag of other fmtp that will need to be further defined.
>   MediaCodecParam[] params;
> }
>
> dictionary MediaCodecParam {
>   DOMString key;
>   DOMString value;
> }
>
>
> Notes:
>
> - When LocalMediaStreams are added using addStream, onnegotiatedneeded 
> is not called, and those streams are never reflected in future SDP 
> exchanges.  Indeed, it would be impossible to put them in the SDP 
> without first resolving if that would be Plan A SDP or Plan B SDP.
>
> - Just like piles of attributes would need to be defined for Plan A 
> and for Plan B, similar attributes would need to be defined here 
> (Luckily,  much work has already been done figuring out what those 
> parameters are :).
>
>
> Pros:
>
> - Either Plan A or Plan B or could be implemented in Javascript using 
> this API
> - It exposes all the same functionality to the Javascript as SDP, but 
> in a much nicer format that is much easier to work with.
> - Any other signalling mechanism, such as Jingle or CLUE could be 
> implemented using this API.
> - There is almost no risk of signalling glare.
> - Debugging errors with misconfigured descriptions should be much 
> easier with this than with large SDP blobs.
>
>
> Cons:
>
> - Now there are two slightly different ways to add streams: by 
> creating a LocalMediaStream first, and not.  This is, however, 
> analogous to setting "negotiated: true" in createDataChannel.  On way 
> is "Just Work", and the other is more advanced control.
>
> - All the options in MediaCodecDescription are a bit complicated. 
>  Really, this is only necessary because Plan A requires being able to 
> specify codec parameters per SSRC, and set each flow on different 
> transports.  If we did not have this requirement, we could simplify.
>
>
> Example Usage:
>
> // Imagine I have MyApp, handles creating a PeerConnection,
> // signalling, and rendering streams.  This is how the new API could be
> // used.
> var peerConnection = MyApp.createPeerConnection();
>
> // On sender side:
> var stream = MyApp.getMediaStream();
> var localStream = peerConnection.createSendStream(stream);
> sendStream.description = MyApp.modifyStream(localStream.description)
> MyApp.signalAddStream(localStream.description, function(response)) {
>   if (!response.rejected) {
>     // Media will not be sent.
>     peerConnection.addStream(localStream);
>   }
> }
>
> // On receiver side:
> MyApp.onAddStreamSignalled = function(streamDescription) {
>   var stream = peerConnection.createReceiveStream(streamDescription);
>   MyApp.renderStream(stream);
> }
>
>
> // In this exchange, the MediaStreamDescription signalled from the
> // sender to the receiver may have looked something like this:
>
> {
>   tracks: [
>   {
>     id: "audio1",
>     kind: "audio",
>     flows: [
>     {
> id: "main",
>       transportId: "transport1",
>       ssrc: 1111,
>       codecs: [
>       {
>         payloadType: 111,
>         name: "opus",
>         // ... more codec details
>       },
>       {
>         payloadType: 112,
>         name: "pcmu",
> // ... more codec details
>       }]
>    }]
>  },
>  {
>     id: "video1",
>     kind: "video",
>     flows: [
>     {
>       id: "sim0",
>       transportId: "transport2",
>       ssrc: 2222,
>       codecs: [
>       {
>         payloadType: 122,
>         name: "vp8"
> // ... more codec details
>       }]
>    },
>    {
>      id: "sim1",
>      transportId: "transport2",
>      ssrc: 2223,
>      codecs: [
>      {
>        payloadType: 122,
>        name: "vp8",
> // ... more codec details
>      }]
>    },
>    {
>      id: "sim2",
>      transportId: "transport2",
>      ssrc: 2224,
>      codecs: [
>      {
>        payloadType: 122,
>        name: "vp8",
> // ... more codec details
>      }]
>    },
>
>    {
>      id: "sim0fec",
>      transportId: "transport2",
>      ssrc: 2225,
>      codecs: [
>      {
>        payloadType: 122,
>        name: "vp8",
>        // ...
>      }]
>    }],
>    flowGroups: [
>    {
>      semantics: "SIM",
>      ssrcs: [2222, 2223, 2224]
>    },
>    {
>      semantics: "FEC",
>      ssrcs: [2222, 2225]
>    }]
>  }]
> }
>
>
> Constructive feedback is welcome :).
> _______________________________________________
> rtcweb mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/rtcweb