Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)

Stefan Håkansson LK <stefan.lk.hakansson@ericsson.com> Mon, 17 June 2013 13:58 UTC

Return-Path: <prvs=4880f0f092=stefan.lk.hakansson@ericsson.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BB98621F9BF6 for <rtcweb@ietfa.amsl.com>; Mon, 17 Jun 2013 06:58:47 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.949
X-Spam-Level:
X-Spam-Status: No, score=-5.949 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HELO_EQ_SE=0.35, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4eb2+b7XYkO4 for <rtcweb@ietfa.amsl.com>; Mon, 17 Jun 2013 06:58:41 -0700 (PDT)
Received: from mailgw7.ericsson.se (mailgw7.ericsson.se [193.180.251.48]) by ietfa.amsl.com (Postfix) with ESMTP id 17FC421F9BB8 for <rtcweb@ietf.org>; Mon, 17 Jun 2013 06:58:38 -0700 (PDT)
X-AuditID: c1b4fb30-b7f9e6d000002643-3f-51bf160d41a0
Received: from ESESSHC011.ericsson.se (Unknown_Domain [153.88.253.125]) by mailgw7.ericsson.se (Symantec Mail Security) with SMTP id 3F.11.09795.D061FB15; Mon, 17 Jun 2013 15:58:37 +0200 (CEST)
Received: from ESESSMB209.ericsson.se ([169.254.9.6]) by ESESSHC011.ericsson.se ([153.88.183.51]) with mapi id 14.02.0328.009; Mon, 17 Jun 2013 15:58:36 +0200
From: =?iso-8859-1?Q?Stefan_H=E5kansson_LK?= <stefan.lk.hakansson@ericsson.com>
To: Peter Thatcher <pthatcher@google.com>
Thread-Topic: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)
Thread-Index: AQHOa1pIfKQNX7+x70mEbzKV/FqHgg==
Date: Mon, 17 Jun 2013 13:58:36 +0000
Message-ID: <1447FA0C20ED5147A1AA0EF02890A64B1C2FC071@ESESSMB209.ericsson.se>
References: <CAJrXDUHdoxLTsofiwLBdwBNnCCkCBgjSdbmLaXrNEPODMrsSVA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [153.88.183.16]
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrCLMWRmVeSWpSXmKPExsUyM+JvrS6v2P5Ag7ZVOhbXlr9mtVj7r53d gcljwaZSjyVLfjIFMEVx2yQllpQFZ6bn6dslcGf0bVvIUrAksuL/y7NMDYw7XboYOTkkBEwk 7m97ywZhi0lcuLceyObiEBI4zCjR1fGaEcJZxCgxc1EHI0gVm0CgxNZ9C8A6RAQ0JSZPbmYF sZmB7AnLdgHFOTiEBfIlJjwLBzFFBAokDu1khKjWk+i9MhvMZhFQlWh/MY8dxOYV8JWYuLib CcQWEgiQWDtpDlgNI9A930+tYYKYLi5x68l8Jog7BSSW7DnPDGGLSrx8/I8VwlaU2Hm2nRmi Xk/ixtQpbBC2tsSyha+ZIXYJSpyc+YRlAqPoLCRjZyFpmYWkZRaSlgWMLKsY2XMTM3PSy803 MQIj4eCW3wY7GDfdFzvEKM3BoiTO++nUrkAhgfTEktTs1NSC1KL4otKc1OJDjEwcnCCCS6qB Ufa8zalZ6sGN3fPW3zi7WeOo0NvKVqZL/otv9LrOa4hccfZ1iYo9i0Ke4dJ1/3aoxmtI/9og nlKuf5Or9Y4rL/uuz3JzNA48O+ky74R71cIagYtTwxoSHOZWFZ//NVdj5ZHKBSeaRa0flnqx J3heuTY3w+Jvuu9syW9s+2ImxUtnvgvviIn0V2Ipzkg01GIuKk4EAE/fdAlXAgAA
Cc: "<rtcweb@ietf.org>" <rtcweb@ietf.org>
Subject: Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 17 Jun 2013 13:58:47 -0000

A couple of questions (to help me understand) in-line //Stefan

On 2013-06-17 14:57, Peter Thatcher wrote:
> Google is in full support of "Plan B" for encoding multiple media
> sources in SDP, and would like to see the Plan A vs. Plan B decision
> resolved soon.  Recently, though, a third option, called "NoPlan" has
> been proposed, but it lacked the details of what a JS API would look
> like for NoPlan.  Cullen asked to see such an API proposal, and so I
> have worked with Emil to make one.  He will be adding it to the NoPlan
> draft, but I will also include it in this email.
>
> Again, Google is in full support of "Plan B".  But if Plan A vs. Plan B
> cannot be decided, then we support NoPlan with the following additions
> to the WebRTC JS API as an option that allows implementing either Plan A
> or Plan B  in Javascript.  And even if Plan A vs. Plan B is resolved,
> these API additions would still be a big improvement for those WebRTC
> applications that don't use SDP for signalling.
>
> It is a bit long because I have added many comments and examples, but
> the actually additions only include two new methods on PeerConnection
> and a few new dictionaries.  So don't be overwhelmed :).
>
>
>
> Intro: This follows the model of createDataChannel, which has a JS
> method on PeerConnection that makes it possible to add data channels
> without going through SDP.  Furthermore, just like createDataChannel
> allows 2 ways to handle neogitation (the "I know what I'm doing;  Here's
> what I want to send; Let me signal everything" mode and the "please take
> care of it for me;  send an OPEN message" mode), this also has 2 ways to
> handle negotiation (the "I know what I'm doing; Here's what I want to
> send; Let me signal everything" mode and the "please take care of it for
> me;  send SDP back and forth" mode).
>
> Following the success of createDataChannel, this allows simple
> applications to Just Work and more advanced applications to easily
> control what they need to.  In particular, it's possible to use this API
> to implement either Plan A or Plan B.
>
> // The following two method are added to RTCPeerConnection
> partial interface RTCPeerConnection {
>   // Create a stream that is used to send a source stream.
>   // The MediaSendStream.description can be used for signalling.
>   // No media is sent until addStream(MediaSendStream) is called.
>   LocalMediaStream createLocalStream(MediaStream sourceStream);
>
>   // Create a stream that is used to receive media from the remote side,
>   // given the parameters signalled from MedaiSendStream.description.
>   MediaStream createRemoteStream(MediaStreamDescription description);
> }

what would happen if the application adds or removes a track from 
sourceStream (I'm thinking on how what would impact the created 
LocalMediaStream and the MediaStream at the remote end)?

>
>
> interface LocalMediaStream implements MediaStream {
>    // This can be changed at any time, but especially before calling
>    // PeerConnection.addStream
>    attribute MediaStreamDescription description;
> }
>
>
> // Represents the parameters used to either send or receive a stream
> // over a PeerConnection.
> dictionary MediaStreamDescription {
>    MediaStreamTrackDescription[] tracks;
> }
>
>
> // Represents the parameters used to either send or receive a track over
> // a PeerConnection.  A track has many "flows", which can be grouped
> // together.
> dictionary MediaStreamTrackDescription {
>    // Same as the MediaStreamTrack.id
>    DOMString id;
>
>    // Same as the MediaStreamTrack.kind
>    DOMString kind;
>
>    // A track can have many "flows", such as for Simulcast, FEC, etc.
>    // And they can be grouped in arbitrary ways.
>    MediaFlowDescription[] flows;
>    MediaFlowGroup[] flowGroups;
> }
>
> // Represents the parameters used to either send or receive a "flow"
> // over a PeerConnection.  A "flow" is a media that arrives with a
> // single, unique SSRC.  One to many flows together make up the media
> // for a track.  For example, there may be Simulcast, FEC, and RTX
> // flows.
> dictionay MediaFlowDescription {
>    // The "flow id" must be unique to the track, but need not be unique
>    // outside of the track (two tracks could both have a flow with the
>    // same flow ID).
>    DOMString id;
>
>    // Each flow can go over its own transport.  If the JS sets this to a
>    // transportId that doesn't have a transport setup already, the
>    // browser will use SDP negotiation to setup a transport to back that
>    // transportId.  If This is set to an MID in the SDP, then that MID's
>    // transport is used.
>    DOMString transportId;

Where from do you get transportId (e.g. if you want to re-use one)?

>
>    // The SSRC used to send the flow.
>    unsigned int ssrc;
>
>    // When used as receive parameters, this indicates the possible list
>    // of codecs that might come in for this flow.  For exmample, a given
>    // receive flow could be setup to receive any of OPUS, ISAC, or PCMU.
>    // When used as send parameters, this indicates that the first codec
>    // should be used, but the browser can use send other codecs if it
>    // needs to because of either bandwidth or CPU constraints.
>    MediaCodecDescription[] codecs;
> }
>
>
> dictionary MediaFlowGroup {
>    DOMString type;  // "SIM" for Simulcast, "FEC" for FEC, etc
>    DOMString[] flowids;
> }
>
> dictionary MediaCodecDescription {
>    unsigned byte payloadType;
>    DOMString name;
>    unsigned int? clockRate;
>    unsigned int? bitRate;
>    // A grab bag of other fmtp that will need to be further defined.
>    MediaCodecParam[] params;
> }
>
> dictionary MediaCodecParam {
>    DOMString key;
>    DOMString value;
> }
>
>
> Notes:
>
> - When LocalMediaStreams are added using addStream, onnegotiatedneeded
> is not called, and those streams are never reflected in future SDP
> exchanges.  Indeed, it would be impossible to put them in the SDP
> without first resolving if that would be Plan A SDP or Plan B SDP.
>
> - Just like piles of attributes would need to be defined for Plan A and
> for Plan B, similar attributes would need to be defined here (Luckily,
>   much work has already been done figuring out what those parameters are :).
>
>
> Pros:
>
> - Either Plan A or Plan B or could be implemented in Javascript using
> this API
> - It exposes all the same functionality to the Javascript as SDP, but in
> a much nicer format that is much easier to work with.
> - Any other signalling mechanism, such as Jingle or CLUE could be
> implemented using this API.
> - There is almost no risk of signalling glare.
> - Debugging errors with misconfigured descriptions should be much easier
> with this than with large SDP blobs.
>
>
> Cons:
>
> - Now there are two slightly different ways to add streams: by creating
> a LocalMediaStream first, and not.  This is, however, analogous to
> setting "negotiated: true" in createDataChannel.  On way is "Just Work",
> and the other is more advanced control.
>
> - All the options in MediaCodecDescription are a bit complicated.
>   Really, this is only necessary because Plan A requires being able to
> specify codec parameters per SSRC, and set each flow on different
> transports.  If we did not have this requirement, we could simplify.
>
>
> Example Usage:
>
> // Imagine I have MyApp, handles creating a PeerConnection,
> // signalling, and rendering streams.  This is how the new API could be
> // used.
> var peerConnection = MyApp.createPeerConnection();
>
> // On sender side:
> var stream = MyApp.getMediaStream();
> var localStream = peerConnection.createSendStream(stream);
> sendStream.description = MyApp.modifyStream(localStream.description)

Should it say "localStream.description"?

> MyApp.signalAddStream(localStream.description, function(response)) {
>    if (!response.rejected) {
>      // Media will not be sent.
>      peerConnection.addStream(localStream);
>    }
> }
>
> // On receiver side:
> MyApp.onAddStreamSignalled = function(streamDescription) {
>    var stream = peerConnection.createReceiveStream(streamDescription);
>    MyApp.renderStream(stream);
> }
>
>
> // In this exchange, the MediaStreamDescription signalled from the
> // sender to the receiver may have looked something like this:
>
> {
>    tracks: [
>    {
>      id: "audio1",
>      kind: "audio",
>      flows: [
>      {
> id: "main",
>        transportId: "transport1",
>        ssrc: 1111,
>        codecs: [
>        {
>          payloadType: 111,
>          name: "opus",
>          // ... more codec details
>        },
>        {
>          payloadType: 112,
>          name: "pcmu",
> // ... more codec details
>        }]
>     }]
>   },
>   {
>      id: "video1",
>      kind: "video",
>      flows: [
>      {
>        id: "sim0",
>        transportId: "transport2",
>        ssrc: 2222,
>        codecs: [
>        {
>          payloadType: 122,
>          name: "vp8"
> // ... more codec details
>        }]
>     },
>     {
>       id: "sim1",
>       transportId: "transport2",
>       ssrc: 2223,
>       codecs: [
>       {
>         payloadType: 122,
>         name: "vp8",
> // ... more codec details
>       }]
>     },
>     {
>       id: "sim2",
>       transportId: "transport2",
>       ssrc: 2224,
>       codecs: [
>       {
>         payloadType: 122,
>         name: "vp8",
> // ... more codec details
>       }]
>     },
>
>     {
>       id: "sim0fec",
>       transportId: "transport2",
>       ssrc: 2225,
>       codecs: [
>       {
>         payloadType: 122,
>         name: "vp8",
>         // ...
>       }]
>     }],
>     flowGroups: [
>     {
>       semantics: "SIM",
>       ssrcs: [2222, 2223, 2224]
>     },
>     {
>       semantics: "FEC",
>       ssrcs: [2222, 2225]
>     }]
>   }]
> }
>
>
> Constructive feedback is welcome :).