Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)

[Stefan Håkansson LK <[e-mail]>]

A couple of questions (to help me understand) in-line //Stefan

On 2013-06-17 14:57, Peter Thatcher wrote:
> Google is in full support of "Plan B" for encoding multiple media
> sources in SDP, and would like to see the Plan A vs. Plan B decision
> resolved soon.  Recently, though, a third option, called "NoPlan" has
> been proposed, but it lacked the details of what a JS API would look
> like for NoPlan.  Cullen asked to see such an API proposal, and so I
> have worked with Emil to make one.  He will be adding it to the NoPlan
> draft, but I will also include it in this email.
>
> Again, Google is in full support of "Plan B".  But if Plan A vs. Plan B
> cannot be decided, then we support NoPlan with the following additions
> to the WebRTC JS API as an option that allows implementing either Plan A
> or Plan B  in Javascript.  And even if Plan A vs. Plan B is resolved,
> these API additions would still be a big improvement for those WebRTC
> applications that don't use SDP for signalling.
>
> It is a bit long because I have added many comments and examples, but
> the actually additions only include two new methods on PeerConnection
> and a few new dictionaries.  So don't be overwhelmed :).
>
>
>
> Intro: This follows the model of createDataChannel, which has a JS
> method on PeerConnection that makes it possible to add data channels
> without going through SDP.  Furthermore, just like createDataChannel
> allows 2 ways to handle neogitation (the "I know what I'm doing;  Here's
> what I want to send; Let me signal everything" mode and the "please take
> care of it for me;  send an OPEN message" mode), this also has 2 ways to
> handle negotiation (the "I know what I'm doing; Here's what I want to
> send; Let me signal everything" mode and the "please take care of it for
> me;  send SDP back and forth" mode).
>
> Following the success of createDataChannel, this allows simple
> applications to Just Work and more advanced applications to easily
> control what they need to.  In particular, it's possible to use this API
> to implement either Plan A or Plan B.
>
> // The following two method are added to RTCPeerConnection
> partial interface RTCPeerConnection {
>   // Create a stream that is used to send a source stream.
>   // The MediaSendStream.description can be used for signalling.
>   // No media is sent until addStream(MediaSendStream) is called.
>   LocalMediaStream createLocalStream(MediaStream sourceStream);
>
>   // Create a stream that is used to receive media from the remote side,
>   // given the parameters signalled from MedaiSendStream.description.
>   MediaStream createRemoteStream(MediaStreamDescription description);
> }

what would happen if the application adds or removes a track from 
sourceStream (I'm thinking on how what would impact the created 
LocalMediaStream and the MediaStream at the remote end)?

>
>
> interface LocalMediaStream implements MediaStream {
>    // This can be changed at any time, but especially before calling
>    // PeerConnection.addStream
>    attribute MediaStreamDescription description;
> }
>
>
> // Represents the parameters used to either send or receive a stream
> // over a PeerConnection.
> dictionary MediaStreamDescription {
>    MediaStreamTrackDescription[] tracks;
> }
>
>
> // Represents the parameters used to either send or receive a track over
> // a PeerConnection.  A track has many "flows", which can be grouped
> // together.
> dictionary MediaStreamTrackDescription {
>    // Same as the MediaStreamTrack.id
>    DOMString id;
>
>    // Same as the MediaStreamTrack.kind
>    DOMString kind;
>
>    // A track can have many "flows", such as for Simulcast, FEC, etc.
>    // And they can be grouped in arbitrary ways.
>    MediaFlowDescription[] flows;
>    MediaFlowGroup[] flowGroups;
> }
>
> // Represents the parameters used to either send or receive a "flow"
> // over a PeerConnection.  A "flow" is a media that arrives with a
> // single, unique SSRC.  One to many flows together make up the media
> // for a track.  For example, there may be Simulcast, FEC, and RTX
> // flows.
> dictionay MediaFlowDescription {
>    // The "flow id" must be unique to the track, but need not be unique
>    // outside of the track (two tracks could both have a flow with the
>    // same flow ID).
>    DOMString id;
>
>    // Each flow can go over its own transport.  If the JS sets this to a
>    // transportId that doesn't have a transport setup already, the
>    // browser will use SDP negotiation to setup a transport to back that
>    // transportId.  If This is set to an MID in the SDP, then that MID's
>    // transport is used.
>    DOMString transportId;

Where from do you get transportId (e.g. if you want to re-use one)?

>
>    // The SSRC used to send the flow.
>    unsigned int ssrc;
>
>    // When used as receive parameters, this indicates the possible list
>    // of codecs that might come in for this flow.  For exmample, a given
>    // receive flow could be setup to receive any of OPUS, ISAC, or PCMU.
>    // When used as send parameters, this indicates that the first codec
>    // should be used, but the browser can use send other codecs if it
>    // needs to because of either bandwidth or CPU constraints.
>    MediaCodecDescription[] codecs;
> }
>
>
> dictionary MediaFlowGroup {
>    DOMString type;  // "SIM" for Simulcast, "FEC" for FEC, etc
>    DOMString[] flowids;
> }
>
> dictionary MediaCodecDescription {
>    unsigned byte payloadType;
>    DOMString name;
>    unsigned int? clockRate;
>    unsigned int? bitRate;
>    // A grab bag of other fmtp that will need to be further defined.
>    MediaCodecParam[] params;
> }
>
> dictionary MediaCodecParam {
>    DOMString key;
>    DOMString value;
> }
>
>
> Notes:
>
> - When LocalMediaStreams are added using addStream, onnegotiatedneeded
> is not called, and those streams are never reflected in future SDP
> exchanges.  Indeed, it would be impossible to put them in the SDP
> without first resolving if that would be Plan A SDP or Plan B SDP.
>
> - Just like piles of attributes would need to be defined for Plan A and
> for Plan B, similar attributes would need to be defined here (Luckily,
>   much work has already been done figuring out what those parameters are :).
>
>
> Pros:
>
> - Either Plan A or Plan B or could be implemented in Javascript using
> this API
> - It exposes all the same functionality to the Javascript as SDP, but in
> a much nicer format that is much easier to work with.
> - Any other signalling mechanism, such as Jingle or CLUE could be
> implemented using this API.
> - There is almost no risk of signalling glare.
> - Debugging errors with misconfigured descriptions should be much easier
> with this than with large SDP blobs.
>
>
> Cons:
>
> - Now there are two slightly different ways to add streams: by creating
> a LocalMediaStream first, and not.  This is, however, analogous to
> setting "negotiated: true" in createDataChannel.  On way is "Just Work",
> and the other is more advanced control.
>
> - All the options in MediaCodecDescription are a bit complicated.
>   Really, this is only necessary because Plan A requires being able to
> specify codec parameters per SSRC, and set each flow on different
> transports.  If we did not have this requirement, we could simplify.
>
>
> Example Usage:
>
> // Imagine I have MyApp, handles creating a PeerConnection,
> // signalling, and rendering streams.  This is how the new API could be
> // used.
> var peerConnection = MyApp.createPeerConnection();
>
> // On sender side:
> var stream = MyApp.getMediaStream();
> var localStream = peerConnection.createSendStream(stream);
> sendStream.description = MyApp.modifyStream(localStream.description)

Should it say "localStream.description"?

> MyApp.signalAddStream(localStream.description, function(response)) {
>    if (!response.rejected) {
>      // Media will not be sent.
>      peerConnection.addStream(localStream);
>    }
> }
>
> // On receiver side:
> MyApp.onAddStreamSignalled = function(streamDescription) {
>    var stream = peerConnection.createReceiveStream(streamDescription);
>    MyApp.renderStream(stream);
> }
>
>
> // In this exchange, the MediaStreamDescription signalled from the
> // sender to the receiver may have looked something like this:
>
> {
>    tracks: [
>    {
>      id: "audio1",
>      kind: "audio",
>      flows: [
>      {
> id: "main",
>        transportId: "transport1",
>        ssrc: 1111,
>        codecs: [
>        {
>          payloadType: 111,
>          name: "opus",
>          // ... more codec details
>        },
>        {
>          payloadType: 112,
>          name: "pcmu",
> // ... more codec details
>        }]
>     }]
>   },
>   {
>      id: "video1",
>      kind: "video",
>      flows: [
>      {
>        id: "sim0",
>        transportId: "transport2",
>        ssrc: 2222,
>        codecs: [
>        {
>          payloadType: 122,
>          name: "vp8"
> // ... more codec details
>        }]
>     },
>     {
>       id: "sim1",
>       transportId: "transport2",
>       ssrc: 2223,
>       codecs: [
>       {
>         payloadType: 122,
>         name: "vp8",
> // ... more codec details
>       }]
>     },
>     {
>       id: "sim2",
>       transportId: "transport2",
>       ssrc: 2224,
>       codecs: [
>       {
>         payloadType: 122,
>         name: "vp8",
> // ... more codec details
>       }]
>     },
>
>     {
>       id: "sim0fec",
>       transportId: "transport2",
>       ssrc: 2225,
>       codecs: [
>       {
>         payloadType: 122,
>         name: "vp8",
>         // ...
>       }]
>     }],
>     flowGroups: [
>     {
>       semantics: "SIM",
>       ssrcs: [2222, 2223, 2224]
>     },
>     {
>       semantics: "FEC",
>       ssrcs: [2222, 2225]
>     }]
>   }]
> }
>
>
> Constructive feedback is welcome :).