Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)

[Harald Alvestrand <[e-mail]>]

On 06/18/2013 07:44 AM, Peter Thatcher wrote:
> I need to write a more complete example, but it basically would be:
>
> var signalling = new AppSpecificSignalling();
> var stream = getUserMedia(...);  // For the example, imagine it's 
> synchronous.
> var pc = new PeerConnection(...);
> var dc = pc.createDataChannel(...);
> pc.createOffer(function(offer) {
>   pc.setLocalDescription(offer);
>   signalling.swapTransportParams(offer.sdp, function(answerSdp) {
>     pc.setRemoteDescription(new SessionDescription(answerSdp);
>   });
> });
> var localStream = pc.createLocalStream(stream);
> signalling.addStream(localStream.description);
> signalling.onStreamAdded = function(remoteStreamDescription) {
>   var remoteStream = pc.createRemoteStream(stream);
>   // Render the remoteStream.
> });
>
> And in all of this, the only SDP involved would be to setup the 
> transport the data channel, which would then be used to BUNDLE audio 
> and video over the same (with rtcp-mux, naturally).  Something like this:
>
> v=0 o=- 697639972863421376 2 IN IP4 127.0.0.1
> s=-
> t=0 0
> m=application 1 DTLS/SCTP 5000
> c=IN IP4 0.0.0.0
> a=mid:transport
>
> a=ice-ufrag:tEP42he9r6LycvmM
> a=ice-pwd:cKJLuvHy9pas9rdezUZB9xet
> a=fingerprint:sha-256 
> EC:E0:95:43:B7:01:49:8B:BC:21:CC:41:7F:10:FD:92:28:B8:F8:41:07:70:77:7D:7C:47:92:30:79:C4:95:9C
>
>
>
> 10 lines of SDP (per side) is, relatively speaking, very small, and it 
> really only contains three values: ice-ufrag/ice-pwd, and dtsl 
> fingerprint.
>
> I may be missing something, but I think this would work.  If anyone 
> knows a reason why it wouldn't, I'm sure you'll let me know :).

At the risk of adding some more complexity:

if both sides add the offerToReceiveVideo=true constraint, the SDP will 
also include M-lines for audio and video, indicating supported codecs, 
which allows the initial offer/answer to supply information that allows 
the UA to check that the parameters to the addLocalStream / 
addRemoteStream calls actually make sense in context, that payload type 
numbers actually mean what they're supposed to mean, and so on.

You know, what SDP was originally designed for....

>
>
>
> On Mon, Jun 17, 2013 at 1:07 PM, Jim Barnett 
> <[e-mail] <mailto:[e-mail]>> wrote:
>
>     Peter,
>
>     You say that only initial PeerConnection set up requires offer
>     answer.   Where is that in your example?  In the example, all we
>     see on the receiving side is that it calls createReceiveStream.
>
>     -Jim
>
>     *From:*[e-mail] <mailto:[e-mail]>
>     [mailto:[e-mail] <mailto:[e-mail]>]
>     *On Behalf Of *Peter Thatcher
>     *Sent:* Monday, June 17, 2013 3:26 PM
>     *To:* Martin Thomson
>
>
>     *Cc:* <[e-mail] <mailto:[e-mail]>>
>     *Subject:* Re: [rtcweb] Proposal for a JS API for NoPlan (adding
>     multiple sources without encoding them in SDP)
>
>     Yes, I was expecting you to be more supportive.  I'm surprised out
>     how your want "all or nothing".  I'm afraid if our options for a
>     clean API are all or nothing, we'll just end up with nothing.  I'd
>     prefer to try incremental improvements to word toward (eventually)
>     a clean API.
>
>     Do you think it is impossible to work toward a clean API in an
>     incremental approach?  If you think it's possible, I'd like to
>     hear your thoughts on how.
>
>     By the way, these API additions would greatly minimize the amount
>     of SDP editing necessary for JS clients that don't use SDP for
>     signalling.  And later incremental improvements could reduce it
>     further.  Also, it's no longer necessary to do offer/answer for
>     adding tracks.  It's only the intial PeerConnection setup that
>     needs to do Offer/Answer.  So, it doesn't inherit all the problems
>     quite as much as you described.  It may be slightly abominable,
>     but I certainly consider it less abominable than the SDP editing
>     necessary without it.
>
>     On Mon, Jun 17, 2013 at 11:57 AM, Martin Thomson
>     <[e-mail] <mailto:[e-mail]>> wrote:
>
>     Maybe you'd expect me to be more supportive of something that looked
>     so much like CU-RTC-Web.  It inherits all the worst properties of JSEP
>     (Offer/Answer, SDP editing) with a partial implementation of a clean
>     API.
>
>     It's comment 22-lite.  It's an abomination.  If you are going to do
>     this, do it properly.
>
>
>     On 17 June 2013 05:57, Peter Thatcher <[e-mail]
>     <mailto:[e-mail]>> wrote:
>     > Google is in full support of "Plan B" for encoding multiple
>     media sources in
>     > SDP, and would like to see the Plan A vs. Plan B decision
>     resolved soon.
>     > Recently, though, a third option, called "NoPlan" has been
>     proposed, but it
>     > lacked the details of what a JS API would look like for NoPlan.
>      Cullen
>     > asked to see such an API proposal, and so I have worked with
>     Emil to make
>     > one.  He will be adding it to the NoPlan draft, but I will also
>     include it
>     > in this email.
>     >
>     > Again, Google is in full support of "Plan B".  But if Plan A vs.
>     Plan B
>     > cannot be decided, then we support NoPlan with the following
>     additions to
>     > the WebRTC JS API as an option that allows implementing either
>     Plan A or
>     > Plan B  in Javascript.  And even if Plan A vs. Plan B is
>     resolved, these API
>     > additions would still be a big improvement for those WebRTC
>     applications
>     > that don't use SDP for signalling.
>     >
>     > It is a bit long because I have added many comments and
>     examples, but the
>     > actually additions only include two new methods on
>     PeerConnection and a few
>     > new dictionaries.  So don't be overwhelmed :).
>     >
>     >
>     >
>     > Intro: This follows the model of createDataChannel, which has a
>     JS method on
>     > PeerConnection that makes it possible to add data channels
>     without going
>     > through SDP.  Furthermore, just like createDataChannel allows 2
>     ways to
>     > handle neogitation (the "I know what I'm doing;  Here's what I
>     want to send;
>     > Let me signal everything" mode and the "please take care of it
>     for me;  send
>     > an OPEN message" mode), this also has 2 ways to handle
>     negotiation (the "I
>     > know what I'm doing; Here's what I want to send; Let me signal
>     everything"
>     > mode and the "please take care of it for me;  send SDP back and
>     forth"
>     > mode).
>     >
>     > Following the success of createDataChannel, this allows simple
>     applications
>     > to Just Work and more advanced applications to easily control
>     what they need
>     > to.  In particular, it's possible to use this API to implement
>     either Plan A
>     > or Plan B.
>     >
>     > // The following two method are added to RTCPeerConnection
>     > partial interface RTCPeerConnection {
>     >  // Create a stream that is used to send a source stream.
>     >  // The MediaSendStream.description can be used for signalling.
>     >  // No media is sent until addStream(MediaSendStream) is called.
>     >  LocalMediaStream createLocalStream(MediaStream sourceStream);
>     >
>     >  // Create a stream that is used to receive media from the
>     remote side,
>     >  // given the parameters signalled from MedaiSendStream.description.
>     >  MediaStream createRemoteStream(MediaStreamDescription description);
>     > }
>     >
>     >
>     > interface LocalMediaStream implements MediaStream {
>     >   // This can be changed at any time, but especially before calling
>     >   // PeerConnection.addStream
>     >   attribute MediaStreamDescription description;
>     > }
>     >
>     >
>     > // Represents the parameters used to either send or receive a stream
>     > // over a PeerConnection.
>     > dictionary MediaStreamDescription {
>     >   MediaStreamTrackDescription[] tracks;
>     > }
>     >
>     >
>     > // Represents the parameters used to either send or receive a
>     track over //
>     > a PeerConnection.  A track has many "flows", which can be grouped
>     > // together.
>     > dictionary MediaStreamTrackDescription {
>     >   // Same as the MediaStreamTrack.id
>     >   DOMString id;
>     >
>     >   // Same as the MediaStreamTrack.kind
>     >   DOMString kind;
>     >
>     >   // A track can have many "flows", such as for Simulcast, FEC, etc.
>     >   // And they can be grouped in arbitrary ways.
>     >   MediaFlowDescription[] flows;
>     >   MediaFlowGroup[] flowGroups;
>     > }
>     >
>     > // Represents the parameters used to either send or receive a "flow"
>     > // over a PeerConnection.  A "flow" is a media that arrives with a
>     > // single, unique SSRC.  One to many flows together make up the
>     media
>     > // for a track.  For example, there may be Simulcast, FEC, and RTX
>     > // flows.
>     > dictionay MediaFlowDescription {
>     >   // The "flow id" must be unique to the track, but need not be
>     unique
>     >   // outside of the track (two tracks could both have a flow
>     with the
>     >   // same flow ID).
>     >   DOMString id;
>     >
>     >   // Each flow can go over its own transport.  If the JS sets
>     this to a
>     >   // transportId that doesn't have a transport setup already, the
>     >   // browser will use SDP negotiation to setup a transport to
>     back that
>     >   // transportId.  If This is set to an MID in the SDP, then
>     that MID's
>     >   // transport is used.
>     >   DOMString transportId;
>     >
>     >   // The SSRC used to send the flow.
>     >   unsigned int ssrc;
>     >
>     >   // When used as receive parameters, this indicates the
>     possible list
>     >   // of codecs that might come in for this flow.  For exmample,
>     a given
>     >   // receive flow could be setup to receive any of OPUS, ISAC,
>     or PCMU.
>     >   // When used as send parameters, this indicates that the first
>     codec
>     >   // should be used, but the browser can use send other codecs if it
>     >   // needs to because of either bandwidth or CPU constraints.
>     >   MediaCodecDescription[] codecs;
>     > }
>     >
>     >
>     > dictionary MediaFlowGroup {
>     >   DOMString type;  // "SIM" for Simulcast, "FEC" for FEC, etc
>     >   DOMString[] flowids;
>     > }
>     >
>     > dictionary MediaCodecDescription {
>     >   unsigned byte payloadType;
>     >   DOMString name;
>     >   unsigned int? clockRate;
>     >   unsigned int? bitRate;
>     >   // A grab bag of other fmtp that will need to be further defined.
>     >   MediaCodecParam[] params;
>     > }
>     >
>     > dictionary MediaCodecParam {
>     >   DOMString key;
>     >   DOMString value;
>     > }
>     >
>     >
>     > Notes:
>     >
>     > - When LocalMediaStreams are added using addStream,
>     onnegotiatedneeded is
>     > not called, and those streams are never reflected in future SDP
>     exchanges.
>     > Indeed, it would be impossible to put them in the SDP without first
>     > resolving if that would be Plan A SDP or Plan B SDP.
>     >
>     > - Just like piles of attributes would need to be defined for
>     Plan A and for
>     > Plan B, similar attributes would need to be defined here
>     (Luckily,  much
>     > work has already been done figuring out what those parameters
>     are :).
>     >
>     >
>     > Pros:
>     >
>     > - Either Plan A or Plan B or could be implemented in Javascript
>     using this
>     > API
>     > - It exposes all the same functionality to the Javascript as
>     SDP, but in a
>     > much nicer format that is much easier to work with.
>     > - Any other signalling mechanism, such as Jingle or CLUE could be
>     > implemented using this API.
>     > - There is almost no risk of signalling glare.
>     > - Debugging errors with misconfigured descriptions should be
>     much easier
>     > with this than with large SDP blobs.
>     >
>     >
>     > Cons:
>     >
>     > - Now there are two slightly different ways to add streams: by
>     creating a
>     > LocalMediaStream first, and not.  This is, however, analogous to
>     setting
>     > "negotiated: true" in createDataChannel.  On way is "Just Work",
>     and the
>     > other is more advanced control.
>     >
>     > - All the options in MediaCodecDescription are a bit
>     complicated.  Really,
>     > this is only necessary because Plan A requires being able to
>     specify codec
>     > parameters per SSRC, and set each flow on different transports.
>      If we did
>     > not have this requirement, we could simplify.
>     >
>     >
>     > Example Usage:
>     >
>     > // Imagine I have MyApp, handles creating a PeerConnection,
>     > // signalling, and rendering streams.  This is how the new API
>     could be
>     > // used.
>     > var peerConnection = MyApp.createPeerConnection();
>     >
>     > // On sender side:
>     > var stream = MyApp.getMediaStream();
>     > var localStream = peerConnection.createSendStream(stream);
>     > sendStream.description = MyApp.modifyStream(localStream.description)
>     > MyApp.signalAddStream(localStream.description, function(response)) {
>     >   if (!response.rejected) {
>     >     // Media will not be sent.
>     > peerConnection.addStream(localStream);
>     >   }
>     > }
>     >
>     > // On receiver side:
>     > MyApp.onAddStreamSignalled = function(streamDescription) {
>     >   var stream =
>     peerConnection.createReceiveStream(streamDescription);
>     >   MyApp.renderStream(stream);
>     > }
>     >
>     >
>     > // In this exchange, the MediaStreamDescription signalled from the
>     > // sender to the receiver may have looked something like this:
>     >
>     > {
>     >   tracks: [
>     >   {
>     >     id: "audio1",
>     >     kind: "audio",
>     >     flows: [
>     >     {
>     >       id: "main",
>     >       transportId: "transport1",
>     >       ssrc: 1111,
>     >       codecs: [
>     >       {
>     >         payloadType: 111,
>     >         name: "opus",
>     >         // ... more codec details
>     >       },
>     >       {
>     >         payloadType: 112,
>     >         name: "pcmu",
>     >         // ... more codec details
>     >       }]
>     >    }]
>     >  },
>     >  {
>     >     id: "video1",
>     >     kind: "video",
>     >     flows: [
>     >     {
>     >       id: "sim0",
>     >       transportId: "transport2",
>     >       ssrc: 2222,
>     >       codecs: [
>     >       {
>     >         payloadType: 122,
>     >         name: "vp8"
>     >         // ... more codec details
>     >       }]
>     >    },
>     >    {
>     >      id: "sim1",
>     >      transportId: "transport2",
>     >      ssrc: 2223,
>     >      codecs: [
>     >      {
>     >        payloadType: 122,
>     >        name: "vp8",
>     >        // ... more codec details
>     >      }]
>     >    },
>     >    {
>     >      id: "sim2",
>     >      transportId: "transport2",
>     >      ssrc: 2224,
>     >      codecs: [
>     >      {
>     >        payloadType: 122,
>     >        name: "vp8",
>     >        // ... more codec details
>     >      }]
>     >    },
>     >
>     >    {
>     >      id: "sim0fec",
>     >      transportId: "transport2",
>     >      ssrc: 2225,
>     >      codecs: [
>     >      {
>     >        payloadType: 122,
>     >        name: "vp8",
>     >        // ...
>     >      }]
>     >    }],
>     >    flowGroups: [
>     >    {
>     >      semantics: "SIM",
>     >      ssrcs: [2222, 2223, 2224]
>     >    },
>     >    {
>     >      semantics: "FEC",
>     >      ssrcs: [2222, 2225]
>     >    }]
>     >  }]
>     > }
>     >
>     >
>     > Constructive feedback is welcome :).
>     >
>
>     > _______________________________________________
>     > rtcweb mailing list
>     > [e-mail] <mailto:[e-mail]>
>     > https://www.ietf.org/mailman/listinfo/rtcweb
>     >
>
>
>
>
> _______________________________________________
> rtcweb mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/rtcweb