Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)

[Peter Thatcher <[e-mail]>]

On Mon, Jun 17, 2013 at 8:27 PM, Silvia Pfeiffer
<[e-mail]>wrote:

>
>
> On Tue, Jun 18, 2013 at 12:54 PM, Robin Raymond <[e-mail]>wrote:
>
>>
>> I actually need access to the raw controls like SSRC, detailed ICE
>> candidate control, encryption keying, etc, and as much control over how
>> media should be controlled/behave as possible. Those things are extremely
>> important for interoperability and signaling and have specific meaning for
>> those who deal with that kind of stuff daily.
>>
>
> Sure - but if you need to hack SDP directly, you're already able to do
> that.
>
>
>
What we'd like is the same power as before (setting all these parameters)
without parsing SDP and serializing SDP.  My proposal allows (optionally)
for just that for MediaStreams, but not for transports.


>  However, as a JavaScript developer, you likely need a simple high level
>> API. Why should you care about all that low level junk? It's meaningless
>> nonsense to you.
>>
>> Real-Time Communication is hard because there is a lot involved,
>> especially if there is any kind of compatibility, advanced features or
>> network control. Nobody wants a difficult API for the sake of being
>> difficult, especially for a sleek language like JavaScript. It's more that
>> it's just necessary if we audio/video people want to do some really cool
>> features that work with new or existing platforms, devices and services.
>>
>> But we can have the best of both worlds!
>>
>
> Agreed, that's what I'm after, but not as a library, but rather as part of
> the WebRTC API.
>

I think the best the WebRTC API can do is make a clean API at the "right"
abstraction level and then let libraries build on that.  For example, a
Jingle library or a p2p mesh library or a multiway video library.  We can't
expect the API to be all things to all people, nor can we expect it to be
so high level that it precludes many useful use cases.


>
>

>
>> There are many people who are in this RTC domain who can wrap that lower
>> level API to give very simple and easy to understand conceptual APIs that
>> abstract that weird funky stuff away. I'd be one of those people to offer
>> such a simpler library (along with many others I'm sure).
>>
>
> I think you should write a proposal!
>
>
>> In some way, it's not all that dissimilar with DOM and CSS3, etc.; jQuery
>> and jQuery UI (and other libraries) create wrappers to make access and
>> control easier but for those who need to raw control they have it. Most
>> people use wrappers and toolkits when they are trying to do the standard
>> stuff, but someone put together the APIs originally to make common use
>> cases.
>>
>
> jQuery was necessary because HTML did not evolve. Now that a lot of the
> jQuery APIs have been added to HTML directly, you can even get away pretty
> well without using jQuery.
>
> On a side note, I agree that SDP should go (and quickly) but it's not the
>> format that's the problem. Granted, it is obtuse. But it's not just that;
>> it's a monolithic do-everything offer/answer model that offers very little
>> control and the API to control the browser's RTC is effectively via
>> manipulation of the SDP. Yuck! Even if it were fancier and prettier JSON,
>> it would still be an ugly do-everything monolithic object with a sketchy
>> offer/answer model that is brittle and offered very little real-scenario
>> controls for those whom need it. It's absolutely horrible and is in good
>> need of quick deprecation.
>>
>
> Right - by providing a higher level abstraction of it, you'd enable that
> stuff underneath to be replaced. I don't have a good handle on SDP, so but
> I think what is proposed can be improved. If you have any ideas on how to,
> this would be a good time to help out, IMHO.
>

I think we can split the discussion of APIs into two separate parts: media
streams and transports.  For media streams, I think what I propose allows
for JS apps to handle MediaStreams without SDP and without O/A, while
working well with what we already have working.  It's not just pretty JSON.
 It gives what I consider a clean API at approximately the "right" level of
abstraction for MediaSteams.


The other half of the equation is transports.  I have some ideas on how
that could be done in a clean API at the "right" abstraction, but I haven't
proposed anything yet specifically because I don't want to blow everything
up at once.  I'd like to try and accomplish this incrementally.


> Cheers,
> Silvia.
>
>
>> -Robin
>>
>>
>>
>>
>>    Silvia Pfeiffer <[e-mail]>
>>  17 June, 2013 9:00 PM
>> Hi Peter, all,
>>
>> I'm looking at all this from the view of a JS developer and I am
>> really excited that there is movement in this space. Having hit my
>> head hard against the limitations of the current WebRTC API and being
>> forced to learn SDP to achieve some of the less common use cases, I'm
>> feeling the pain. I am therefore happy to see that there is a proposal
>> for a higher-level API with similarities to the Microsoft's CU-WebRTC
>> proposal and I hope that together with Microsoft's input this can
>> become a really useful API.
>>
>> What I would like to see, though, is a bit different from what you've
>> proposed. In particular, the MediaFlowDescription object is the one
>> object that I believe is supposed to enable JS developers to do "SDP
>> hacking" without having to understand SDP. Unfortunately, the way in
>> which it is currently written, this API doesn't help a JS developer
>> much. There are properties in that object like "ssrc" that mean
>> nothing unless you understand SDP.
>>
>> I would therefore like to recommend making the properties on the
>> MediaFlowDescription more semantic - in particular giving them better
>> names - such that a JS developer really doesn't have to understand SDP
>> to create/change media flow descriptions. Can we find better names for
>> id, transportId and ssrc that are more explicitly expressing what
>> they stand for and when a JS developer would actually change them?
>>
>> It would be nice if the MediaFlowDescription was abstract enough such
>> that in future it doesn't matter if SDP is actually underneath (with
>> offer/answer), but that's not actually my main goal. What I'm after is
>> an API that can be used without having to understand what is
>> underneath.
>>
>> Thanks for listening and HTH,
>> Silvia.
>>
>> _______________________________________________
>> rtcweb mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/rtcweb
>>    Peter Thatcher <[e-mail]>
>>  17 June, 2013 8:57 AM
>> Google is in full support of "Plan B" for encoding multiple media sources
>> in SDP, and would like to see the Plan A vs. Plan B decision resolved soon.
>>  Recently, though, a third option, called "NoPlan" has been proposed, but
>> it lacked the details of what a JS API would look like for NoPlan.  Cullen
>> asked to see such an API proposal, and so I have worked with Emil to make
>> one.  He will be adding it to the NoPlan draft, but I will also include it
>> in this email.
>>
>> Again, Google is in full support of "Plan B".  But if Plan A vs. Plan B
>> cannot be decided, then we support NoPlan with the following additions to
>> the WebRTC JS API as an option that allows implementing either Plan A or
>> Plan B  in Javascript.  And even if Plan A vs. Plan B is resolved, these
>> API additions would still be a big improvement for those WebRTC
>> applications that don't use SDP for signalling.
>>
>> It is a bit long because I have added many comments and examples, but the
>> actually additions only include two new methods on PeerConnection and a few
>> new dictionaries.  So don't be overwhelmed :).
>>
>>
>>
>> Intro: This follows the model of createDataChannel, which has a JS method
>> on PeerConnection that makes it possible to add data channels without going
>> through SDP.  Furthermore, just like createDataChannel allows 2 ways to
>> handle neogitation (the "I know what I'm doing;  Here's what I want to
>> send; Let me signal everything" mode and the "please take care of it for
>> me;  send an OPEN message" mode), this also has 2 ways to handle
>> negotiation (the "I know what I'm doing; Here's what I want to send; Let me
>> signal everything" mode and the "please take care of it for me;  send SDP
>> back and forth" mode).
>>
>> Following the success of createDataChannel, this allows simple
>> applications to Just Work and more advanced applications to easily control
>> what they need to.  In particular, it's possible to use this API to
>> implement either Plan A or Plan B.
>>
>> // The following two method are added to RTCPeerConnection
>> partial interface RTCPeerConnection {
>>  // Create a stream that is used to send a source stream.
>>  // The MediaSendStream.description can be used for signalling.
>>  // No media is sent until addStream(MediaSendStream) is called.
>>  LocalMediaStream createLocalStream(MediaStream sourceStream);
>>
>>  // Create a stream that is used to receive media from the remote side,
>>  // given the parameters signalled from MedaiSendStream.description.
>>  MediaStream createRemoteStream(MediaStreamDescription description);
>> }
>>
>>
>> interface LocalMediaStream implements MediaStream {
>>   // This can be changed at any time, but especially before calling
>>   // PeerConnection.addStream
>>   attribute MediaStreamDescription description;
>> }
>>
>>
>> // Represents the parameters used to either send or receive a stream
>> // over a PeerConnection.
>> dictionary MediaStreamDescription {
>>   MediaStreamTrackDescription[] tracks;
>> }
>>
>>
>> // Represents the parameters used to either send or receive a track over
>> // a PeerConnection.  A track has many "flows", which can be grouped
>> // together.
>> dictionary MediaStreamTrackDescription {
>>   // Same as the MediaStreamTrack.id
>>   DOMString id;
>>
>>   // Same as the MediaStreamTrack.kind
>>   DOMString kind;
>>
>>   // A track can have many "flows", such as for Simulcast, FEC, etc.
>>   // And they can be grouped in arbitrary ways.
>>   MediaFlowDescription[] flows;
>>   MediaFlowGroup[] flowGroups;
>> }
>>
>> // Represents the parameters used to either send or receive a "flow"
>> // over a PeerConnection.  A "flow" is a media that arrives with a
>> // single, unique SSRC.  One to many flows together make up the media
>> // for a track.  For example, there may be Simulcast, FEC, and RTX
>> // flows.
>> dictionay MediaFlowDescription {
>>   // The "flow id" must be unique to the track, but need not be unique
>>   // outside of the track (two tracks could both have a flow with the
>>   // same flow ID).
>>   DOMString id;
>>
>>   // Each flow can go over its own transport.  If the JS sets this to a
>>   // transportId that doesn't have a transport setup already, the
>>   // browser will use SDP negotiation to setup a transport to back that
>>   // transportId.  If This is set to an MID in the SDP, then that MID's
>>   // transport is used.
>>   DOMString transportId;
>>
>>   // The SSRC used to send the flow.
>>   unsigned int ssrc;
>>
>>   // When used as receive parameters, this indicates the possible list
>>   // of codecs that might come in for this flow.  For exmample, a given
>>   // receive flow could be setup to receive any of OPUS, ISAC, or PCMU.
>>   // When used as send parameters, this indicates that the first codec
>>   // should be used, but the browser can use send other codecs if it
>>   // needs to because of either bandwidth or CPU constraints.
>>   MediaCodecDescription[] codecs;
>> }
>>
>>
>> dictionary MediaFlowGroup {
>>   DOMString type;  // "SIM" for Simulcast, "FEC" for FEC, etc
>>   DOMString[] flowids;
>> }
>>
>> dictionary MediaCodecDescription {
>>   unsigned byte payloadType;
>>   DOMString name;
>>   unsigned int? clockRate;
>>   unsigned int? bitRate;
>>   // A grab bag of other fmtp that will need to be further defined.
>>   MediaCodecParam[] params;
>> }
>>
>> dictionary MediaCodecParam {
>>   DOMString key;
>>   DOMString value;
>> }
>>
>>
>> Notes:
>>
>> - When LocalMediaStreams are added using addStream, onnegotiatedneeded is
>> not called, and those streams are never reflected in future SDP exchanges.
>>  Indeed, it would be impossible to put them in the SDP without first
>> resolving if that would be Plan A SDP or Plan B SDP.
>>
>> - Just like piles of attributes would need to be defined for Plan A and
>> for Plan B, similar attributes would need to be defined here (Luckily,
>>  much work has already been done figuring out what those parameters are :).
>>
>>
>> Pros:
>>
>> - Either Plan A or Plan B or could be implemented in Javascript using
>> this API
>> - It exposes all the same functionality to the Javascript as SDP, but in
>> a much nicer format that is much easier to work with.
>> - Any other signalling mechanism, such as Jingle or CLUE could be
>> implemented using this API.
>> - There is almost no risk of signalling glare.
>> - Debugging errors with misconfigured descriptions should be much easier
>> with this than with large SDP blobs.
>>
>>
>> Cons:
>>
>> - Now there are two slightly different ways to add streams: by creating a
>> LocalMediaStream first, and not.  This is, however, analogous to setting
>> "negotiated: true" in createDataChannel.  On way is "Just Work", and the
>> other is more advanced control.
>>
>> - All the options in MediaCodecDescription are a bit complicated.
>>  Really, this is only necessary because Plan A requires being able to
>> specify codec parameters per SSRC, and set each flow on different
>> transports.  If we did not have this requirement, we could simplify.
>>
>>
>> Example Usage:
>>
>> // Imagine I have MyApp, handles creating a PeerConnection,
>> // signalling, and rendering streams.  This is how the new API could be
>> // used.
>> var peerConnection = MyApp.createPeerConnection();
>>
>> // On sender side:
>> var stream = MyApp.getMediaStream();
>> var localStream = peerConnection.createSendStream(stream);
>> sendStream.description = MyApp.modifyStream(localStream.description)
>> MyApp.signalAddStream(localStream.description, function(response)) {
>>   if (!response.rejected) {
>>     // Media will not be sent.
>>     peerConnection.addStream(localStream);
>>   }
>> }
>>
>> // On receiver side:
>> MyApp.onAddStreamSignalled = function(streamDescription) {
>>   var stream = peerConnection.createReceiveStream(streamDescription);
>>   MyApp.renderStream(stream);
>> }
>>
>>
>> // In this exchange, the MediaStreamDescription signalled from the
>> // sender to the receiver may have looked something like this:
>>
>> {
>>   tracks: [
>>   {
>>     id: "audio1",
>>     kind: "audio",
>>     flows: [
>>     {
>>       id: "main",
>>       transportId: "transport1",
>>       ssrc: 1111,
>>       codecs: [
>>       {
>>         payloadType: 111,
>>         name: "opus",
>>         // ... more codec details
>>       },
>>       {
>>         payloadType: 112,
>>         name: "pcmu",
>>         // ... more codec details
>>       }]
>>    }]
>>  },
>>  {
>>     id: "video1",
>>     kind: "video",
>>     flows: [
>>     {
>>       id: "sim0",
>>       transportId: "transport2",
>>       ssrc: 2222,
>>       codecs: [
>>       {
>>         payloadType: 122,
>>         name: "vp8"
>>         // ... more codec details
>>       }]
>>    },
>>    {
>>      id: "sim1",
>>      transportId: "transport2",
>>      ssrc: 2223,
>>      codecs: [
>>      {
>>        payloadType: 122,
>>        name: "vp8",
>>        // ... more codec details
>>      }]
>>    },
>>    {
>>      id: "sim2",
>>      transportId: "transport2",
>>      ssrc: 2224,
>>      codecs: [
>>      {
>>        payloadType: 122,
>>        name: "vp8",
>>        // ... more codec details
>>      }]
>>    },
>>
>>    {
>>      id: "sim0fec",
>>      transportId: "transport2",
>>      ssrc: 2225,
>>      codecs: [
>>      {
>>        payloadType: 122,
>>        name: "vp8",
>>        // ...
>>      }]
>>    }],
>>    flowGroups: [
>>    {
>>      semantics: "SIM",
>>      ssrcs: [2222, 2223, 2224]
>>    },
>>    {
>>      semantics: "FEC",
>>      ssrcs: [2222, 2225]
>>    }]
>>  }]
>> }
>>
>>
>> Constructive feedback is welcome :).
>> _______________________________________________
>> rtcweb mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/rtcweb
>>
>>
>