Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)

[Silvia Pfeiffer <[e-mail]>]

On Tue, Jun 18, 2013 at 12:54 PM, Robin Raymond <[e-mail]> wrote:

>
> I actually need access to the raw controls like SSRC, detailed ICE
> candidate control, encryption keying, etc, and as much control over how
> media should be controlled/behave as possible. Those things are extremely
> important for interoperability and signaling and have specific meaning for
> those who deal with that kind of stuff daily.
>

Sure - but if you need to hack SDP directly, you're already able to do that.


However, as a JavaScript developer, you likely need a simple high level
> API. Why should you care about all that low level junk? It's meaningless
> nonsense to you.
>
> Real-Time Communication is hard because there is a lot involved,
> especially if there is any kind of compatibility, advanced features or
> network control. Nobody wants a difficult API for the sake of being
> difficult, especially for a sleek language like JavaScript. It's more that
> it's just necessary if we audio/video people want to do some really cool
> features that work with new or existing platforms, devices and services.
>
> But we can have the best of both worlds!
>

Agreed, that's what I'm after, but not as a library, but rather as part of
the WebRTC API.


> There are many people who are in this RTC domain who can wrap that lower
> level API to give very simple and easy to understand conceptual APIs that
> abstract that weird funky stuff away. I'd be one of those people to offer
> such a simpler library (along with many others I'm sure).
>

I think you should write a proposal!


> In some way, it's not all that dissimilar with DOM and CSS3, etc.; jQuery
> and jQuery UI (and other libraries) create wrappers to make access and
> control easier but for those who need to raw control they have it. Most
> people use wrappers and toolkits when they are trying to do the standard
> stuff, but someone put together the APIs originally to make common use
> cases.
>

jQuery was necessary because HTML did not evolve. Now that a lot of the
jQuery APIs have been added to HTML directly, you can even get away pretty
well without using jQuery.

On a side note, I agree that SDP should go (and quickly) but it's not the
> format that's the problem. Granted, it is obtuse. But it's not just that;
> it's a monolithic do-everything offer/answer model that offers very little
> control and the API to control the browser's RTC is effectively via
> manipulation of the SDP. Yuck! Even if it were fancier and prettier JSON,
> it would still be an ugly do-everything monolithic object with a sketchy
> offer/answer model that is brittle and offered very little real-scenario
> controls for those whom need it. It's absolutely horrible and is in good
> need of quick deprecation.
>

Right - by providing a higher level abstraction of it, you'd enable that
stuff underneath to be replaced. I don't have a good handle on SDP, so but
I think what is proposed can be improved. If you have any ideas on how to,
this would be a good time to help out, IMHO.

Cheers,
Silvia.


> -Robin
>
>
>
>
>   Silvia Pfeiffer <[e-mail]>
>  17 June, 2013 9:00 PM
> Hi Peter, all,
>
> I'm looking at all this from the view of a JS developer and I am
> really excited that there is movement in this space. Having hit my
> head hard against the limitations of the current WebRTC API and being
> forced to learn SDP to achieve some of the less common use cases, I'm
> feeling the pain. I am therefore happy to see that there is a proposal
> for a higher-level API with similarities to the Microsoft's CU-WebRTC
> proposal and I hope that together with Microsoft's input this can
> become a really useful API.
>
> What I would like to see, though, is a bit different from what you've
> proposed. In particular, the MediaFlowDescription object is the one
> object that I believe is supposed to enable JS developers to do "SDP
> hacking" without having to understand SDP. Unfortunately, the way in
> which it is currently written, this API doesn't help a JS developer
> much. There are properties in that object like "ssrc" that mean
> nothing unless you understand SDP.
>
> I would therefore like to recommend making the properties on the
> MediaFlowDescription more semantic - in particular giving them better
> names - such that a JS developer really doesn't have to understand SDP
> to create/change media flow descriptions. Can we find better names for
> id, transportId and ssrc that are more explicitly expressing what
> they stand for and when a JS developer would actually change them?
>
> It would be nice if the MediaFlowDescription was abstract enough such
> that in future it doesn't matter if SDP is actually underneath (with
> offer/answer), but that's not actually my main goal. What I'm after is
> an API that can be used without having to understand what is
> underneath.
>
> Thanks for listening and HTH,
> Silvia.
>
> _______________________________________________
> rtcweb mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/rtcweb
>   Peter Thatcher <[e-mail]>
>  17 June, 2013 8:57 AM
> Google is in full support of "Plan B" for encoding multiple media sources
> in SDP, and would like to see the Plan A vs. Plan B decision resolved soon.
>  Recently, though, a third option, called "NoPlan" has been proposed, but
> it lacked the details of what a JS API would look like for NoPlan.  Cullen
> asked to see such an API proposal, and so I have worked with Emil to make
> one.  He will be adding it to the NoPlan draft, but I will also include it
> in this email.
>
> Again, Google is in full support of "Plan B".  But if Plan A vs. Plan B
> cannot be decided, then we support NoPlan with the following additions to
> the WebRTC JS API as an option that allows implementing either Plan A or
> Plan B  in Javascript.  And even if Plan A vs. Plan B is resolved, these
> API additions would still be a big improvement for those WebRTC
> applications that don't use SDP for signalling.
>
> It is a bit long because I have added many comments and examples, but the
> actually additions only include two new methods on PeerConnection and a few
> new dictionaries.  So don't be overwhelmed :).
>
>
>
> Intro: This follows the model of createDataChannel, which has a JS method
> on PeerConnection that makes it possible to add data channels without going
> through SDP.  Furthermore, just like createDataChannel allows 2 ways to
> handle neogitation (the "I know what I'm doing;  Here's what I want to
> send; Let me signal everything" mode and the "please take care of it for
> me;  send an OPEN message" mode), this also has 2 ways to handle
> negotiation (the "I know what I'm doing; Here's what I want to send; Let me
> signal everything" mode and the "please take care of it for me;  send SDP
> back and forth" mode).
>
> Following the success of createDataChannel, this allows simple
> applications to Just Work and more advanced applications to easily control
> what they need to.  In particular, it's possible to use this API to
> implement either Plan A or Plan B.
>
> // The following two method are added to RTCPeerConnection
> partial interface RTCPeerConnection {
>  // Create a stream that is used to send a source stream.
>  // The MediaSendStream.description can be used for signalling.
>  // No media is sent until addStream(MediaSendStream) is called.
>  LocalMediaStream createLocalStream(MediaStream sourceStream);
>
>  // Create a stream that is used to receive media from the remote side,
>  // given the parameters signalled from MedaiSendStream.description.
>  MediaStream createRemoteStream(MediaStreamDescription description);
> }
>
>
> interface LocalMediaStream implements MediaStream {
>   // This can be changed at any time, but especially before calling
>   // PeerConnection.addStream
>   attribute MediaStreamDescription description;
> }
>
>
> // Represents the parameters used to either send or receive a stream
> // over a PeerConnection.
> dictionary MediaStreamDescription {
>   MediaStreamTrackDescription[] tracks;
> }
>
>
> // Represents the parameters used to either send or receive a track over
> // a PeerConnection.  A track has many "flows", which can be grouped
> // together.
> dictionary MediaStreamTrackDescription {
>   // Same as the MediaStreamTrack.id
>   DOMString id;
>
>   // Same as the MediaStreamTrack.kind
>   DOMString kind;
>
>   // A track can have many "flows", such as for Simulcast, FEC, etc.
>   // And they can be grouped in arbitrary ways.
>   MediaFlowDescription[] flows;
>   MediaFlowGroup[] flowGroups;
> }
>
> // Represents the parameters used to either send or receive a "flow"
> // over a PeerConnection.  A "flow" is a media that arrives with a
> // single, unique SSRC.  One to many flows together make up the media
> // for a track.  For example, there may be Simulcast, FEC, and RTX
> // flows.
> dictionay MediaFlowDescription {
>   // The "flow id" must be unique to the track, but need not be unique
>   // outside of the track (two tracks could both have a flow with the
>   // same flow ID).
>   DOMString id;
>
>   // Each flow can go over its own transport.  If the JS sets this to a
>   // transportId that doesn't have a transport setup already, the
>   // browser will use SDP negotiation to setup a transport to back that
>   // transportId.  If This is set to an MID in the SDP, then that MID's
>   // transport is used.
>   DOMString transportId;
>
>   // The SSRC used to send the flow.
>   unsigned int ssrc;
>
>   // When used as receive parameters, this indicates the possible list
>   // of codecs that might come in for this flow.  For exmample, a given
>   // receive flow could be setup to receive any of OPUS, ISAC, or PCMU.
>   // When used as send parameters, this indicates that the first codec
>   // should be used, but the browser can use send other codecs if it
>   // needs to because of either bandwidth or CPU constraints.
>   MediaCodecDescription[] codecs;
> }
>
>
> dictionary MediaFlowGroup {
>   DOMString type;  // "SIM" for Simulcast, "FEC" for FEC, etc
>   DOMString[] flowids;
> }
>
> dictionary MediaCodecDescription {
>   unsigned byte payloadType;
>   DOMString name;
>   unsigned int? clockRate;
>   unsigned int? bitRate;
>   // A grab bag of other fmtp that will need to be further defined.
>   MediaCodecParam[] params;
> }
>
> dictionary MediaCodecParam {
>   DOMString key;
>   DOMString value;
> }
>
>
> Notes:
>
> - When LocalMediaStreams are added using addStream, onnegotiatedneeded is
> not called, and those streams are never reflected in future SDP exchanges.
>  Indeed, it would be impossible to put them in the SDP without first
> resolving if that would be Plan A SDP or Plan B SDP.
>
> - Just like piles of attributes would need to be defined for Plan A and
> for Plan B, similar attributes would need to be defined here (Luckily,
>  much work has already been done figuring out what those parameters are :).
>
>
> Pros:
>
> - Either Plan A or Plan B or could be implemented in Javascript using this
> API
> - It exposes all the same functionality to the Javascript as SDP, but in a
> much nicer format that is much easier to work with.
> - Any other signalling mechanism, such as Jingle or CLUE could be
> implemented using this API.
> - There is almost no risk of signalling glare.
> - Debugging errors with misconfigured descriptions should be much easier
> with this than with large SDP blobs.
>
>
> Cons:
>
> - Now there are two slightly different ways to add streams: by creating a
> LocalMediaStream first, and not.  This is, however, analogous to setting
> "negotiated: true" in createDataChannel.  On way is "Just Work", and the
> other is more advanced control.
>
> - All the options in MediaCodecDescription are a bit complicated.  Really,
> this is only necessary because Plan A requires being able to specify codec
> parameters per SSRC, and set each flow on different transports.  If we did
> not have this requirement, we could simplify.
>
>
> Example Usage:
>
> // Imagine I have MyApp, handles creating a PeerConnection,
> // signalling, and rendering streams.  This is how the new API could be
> // used.
> var peerConnection = MyApp.createPeerConnection();
>
> // On sender side:
> var stream = MyApp.getMediaStream();
> var localStream = peerConnection.createSendStream(stream);
> sendStream.description = MyApp.modifyStream(localStream.description)
> MyApp.signalAddStream(localStream.description, function(response)) {
>   if (!response.rejected) {
>     // Media will not be sent.
>     peerConnection.addStream(localStream);
>   }
> }
>
> // On receiver side:
> MyApp.onAddStreamSignalled = function(streamDescription) {
>   var stream = peerConnection.createReceiveStream(streamDescription);
>   MyApp.renderStream(stream);
> }
>
>
> // In this exchange, the MediaStreamDescription signalled from the
> // sender to the receiver may have looked something like this:
>
> {
>   tracks: [
>   {
>     id: "audio1",
>     kind: "audio",
>     flows: [
>     {
>       id: "main",
>       transportId: "transport1",
>       ssrc: 1111,
>       codecs: [
>       {
>         payloadType: 111,
>         name: "opus",
>         // ... more codec details
>       },
>       {
>         payloadType: 112,
>         name: "pcmu",
>         // ... more codec details
>       }]
>    }]
>  },
>  {
>     id: "video1",
>     kind: "video",
>     flows: [
>     {
>       id: "sim0",
>       transportId: "transport2",
>       ssrc: 2222,
>       codecs: [
>       {
>         payloadType: 122,
>         name: "vp8"
>         // ... more codec details
>       }]
>    },
>    {
>      id: "sim1",
>      transportId: "transport2",
>      ssrc: 2223,
>      codecs: [
>      {
>        payloadType: 122,
>        name: "vp8",
>        // ... more codec details
>      }]
>    },
>    {
>      id: "sim2",
>      transportId: "transport2",
>      ssrc: 2224,
>      codecs: [
>      {
>        payloadType: 122,
>        name: "vp8",
>        // ... more codec details
>      }]
>    },
>
>    {
>      id: "sim0fec",
>      transportId: "transport2",
>      ssrc: 2225,
>      codecs: [
>      {
>        payloadType: 122,
>        name: "vp8",
>        // ...
>      }]
>    }],
>    flowGroups: [
>    {
>      semantics: "SIM",
>      ssrcs: [2222, 2223, 2224]
>    },
>    {
>      semantics: "FEC",
>      ssrcs: [2222, 2225]
>    }]
>  }]
> }
>
>
> Constructive feedback is welcome :).
> _______________________________________________
> rtcweb mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/rtcweb
>
>