Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)

Stefan Håkansson LK <stefan.lk.hakansson@ericsson.com> Tue, 18 June 2013 08:27 UTC

Return-Path: <prvs=4881ea1215=stefan.lk.hakansson@ericsson.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D305521F9CA8 for <rtcweb@ietfa.amsl.com>; Tue, 18 Jun 2013 01:27:55 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.124
X-Spam-Level:
X-Spam-Status: No, score=-4.124 tagged_above=-999 required=5 tests=[AWL=-1.825, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cAJ1xj6gK8IB for <rtcweb@ietfa.amsl.com>; Tue, 18 Jun 2013 01:27:50 -0700 (PDT)
Received: from sesbmg20.ericsson.net (sesbmg20.ericsson.net [193.180.251.56]) by ietfa.amsl.com (Postfix) with ESMTP id C748821F9CB7 for <rtcweb@ietf.org>; Tue, 18 Jun 2013 01:27:48 -0700 (PDT)
X-AuditID: c1b4fb38-b7fa16d0000027dd-57-51c01a0375cc
Received: from ESESSHC003.ericsson.se (Unknown_Domain [153.88.253.125]) by sesbmg20.ericsson.net (Symantec Mail Security) with SMTP id 39.ED.10205.30A10C15; Tue, 18 Jun 2013 10:27:47 +0200 (CEST)
Received: from ESESSMB209.ericsson.se ([169.254.9.6]) by ESESSHC003.ericsson.se ([153.88.183.27]) with mapi id 14.02.0328.009; Tue, 18 Jun 2013 10:27:46 +0200
From: =?Windows-1252?Q?Stefan_H=E5kansson_LK?= <stefan.lk.hakansson@ericsson.com>
To: Peter Thatcher <pthatcher@google.com>
Thread-Topic: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)
Thread-Index: AQHOa1pIfKQNX7+x70mEbzKV/FqHgg==
Date: Tue, 18 Jun 2013 08:27:46 +0000
Message-ID: <1447FA0C20ED5147A1AA0EF02890A64B1C2FC3F0@ESESSMB209.ericsson.se>
References: <CAJrXDUHdoxLTsofiwLBdwBNnCCkCBgjSdbmLaXrNEPODMrsSVA@mail.gmail.com> <1447FA0C20ED5147A1AA0EF02890A64B1C2FC071@ESESSMB209.ericsson.se> <CAJrXDUEL1ynci_bwLydYgvWD=+FQKhurzV3vC0X4LrgBjUMrkA@mail.gmail.com> <57A15FAF9E58F841B2B1651FFE16D28104C918@GENSJZMBX01.msg.int.genesyslab.com> <CAJrXDUHphQq+=vWRNjuRN1BAcUz4kfUf7DEwCPgBBX-esQh0HQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [153.88.183.16]
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrHLMWRmVeSWpSXmKPExsUyM+JvrS6z1IFAg+6H4hZNV6exWFxb/prV Yu2/dnYHZo+lbzvYPBZsKvVYsuQnUwBzFLdNUmJJWXBmep6+XQJ3xrMJ85gLVvQwVrQvu8/Y wHg2uYuRk0NCwETi/snLTBC2mMSFe+vZQGwhgaOMEv1XbSDsRYwS//96dTFycLAJBEs0/XUD CYsIaEpMntzMCmIzC4RKzDy7hh2kRFggX2LCs3AQU0SgQOLQTkaIaj2J3iuzwWwWAVWJpW82 gy3lFfCVmDOjAWgKF9CiD0wSUz9NBLuAEeia76fWMEGMF5e49WQ+1JUCEkv2nGeGsEUlXj7+ xwphK0rsPNvODFFvIPH+3HwoW1ti2cLXzBDLBCVOznzCMoFRdBaSsbOQtMxC0jILScsCRpZV jBzFqcVJuelGBpsYgdFxcMtvix2Ml//aHGKU5mBREuf9dGpXoJBAemJJanZqakFqUXxRaU5q 8SFGJg5OEMEl1cBY5SP/3U5dfqZK+aTEBbyvjA/xNfEs8DjVz+ndyVzbcPuB4YVVsY7bH/q4 /X6uc+P6KsG3IiIP5Nduvf3/z2vLb9lzFG/b8p/x3pldcWZL1JnlHy5zWIkyNu669V9oMcPU TekbLjlEXjy4z8HyIb/Ey3m1B1jOhUSuM/Z7zBevoDQj0ywh4wqHEktxRqKhFnNRcSIA1KM+ nmECAAA=
Cc: "<rtcweb@ietf.org>" <rtcweb@ietf.org>
Subject: Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 18 Jun 2013 08:27:55 -0000

On 2013-06-17 18:21, Peter Thatcher wrote:
>
>
>
> On Mon, Jun 17, 2013 at 8:21 AM, Jim Barnett <Jim.Barnett@genesyslab.com
> <mailto:Jim.Barnett@genesyslab.com>> wrote:
>
>     What can you modify in a LocalMediaStream by editing its
>     description?  Can you add a video track to it?  That would imply
>     calling gUM, and you’d  have to thread it’s success and failure
>     callbacks up to the editing operation.  Or is the track list fixed,
>     so that you can only tweak transport info by editing the
>     LocalMediaStream’s description?____
>
>     __
>
>
> If the JS calls gUM to get a second MediaStreamTrack (say, for
> screencast), it could simply call createLocalStream again to add it, in
> a different LocalMediaStream, and the remote end would get a second
> MediaStream by calling createRemoteStream.  Why would you want more than
> one video MediaStreamTrack in MediaStream at the same time?

I think there are use-cases for that, especially if keep the notion of 
that sync is maintained within a MediaStream but not across different MS's.

You could easily imagine that you have two cameras capturing the same 
speaker. You'd want lip sync when rendering at a remote peer regardless 
of which video that is played.

>
> In general, before you start sending or receiving, you can edit
> everything side of the MediaStreamTrackDescription:  change the ssrcs,
> codecs, transport, video resolution to send with, what repair flows to
> include, whether to use simulcast, etc.  Basically, anything on a track
> level.
>
> But I don't really see a use case where you would need to add tracks to
> a MediaStreamTrackDescription,

I agree - but that is obvious (a MediaStreamTrackDescription describes 
exactly one MediaStreamTrack). But there are use-cases for adding tracks 
to a MediaStream description.

> other than when parsing signalling and
> building the MediaStreamTrackDescription to send down into
> createRemoteStream.  Perhaps if we had a use case, we could define such
> support.  But otherwise, I'd say it's not allowed.
>
>
> -__Jim____
>
> __ __
>
> *From:*rtcweb-bounces@ietf.org <mailto:rtcweb-bounces@ietf.org>
> [mailto:rtcweb-bounces@ietf.org <mailto:rtcweb-bounces@ietf.org>] *On
> Behalf Of *Peter Thatcher
> *Sent:* Monday, June 17, 2013 11:13 AM
> *To:* Stefan Håkansson LK
> *Cc:* <rtcweb@ietf.org <mailto:rtcweb@ietf.org>>
> *Subject:* Re: [rtcweb] Proposal for a JS API for NoPlan (adding
> multiple sources without encoding them in SDP)____
>
> __ __
>
> Answers:____
>
> __ __
>
> On Mon, Jun 17, 2013 at 6:58 AM, Stefan Håkansson LK
> <stefan.lk.hakansson@ericsson.com
> <mailto:stefan.lk.hakansson@ericsson.com>> wrote:____
>
> A couple of questions (to help me understand) in-line //Stefan____
>
>
> On 2013-06-17 14:57, Peter Thatcher wrote:
>  > Google is in full support of "Plan B" for encoding multiple media
>  > sources in SDP, and would like to see the Plan A vs. Plan B decision
>  > resolved soon.  Recently, though, a third option, called "NoPlan" has
>  > been proposed, but it lacked the details of what a JS API would look
>  > like for NoPlan.  Cullen asked to see such an API proposal, and so I
>  > have worked with Emil to make one.  He will be adding it to the NoPlan
>  > draft, but I will also include it in this email.
>  >
>  > Again, Google is in full support of "Plan B".  But if Plan A vs. Plan B
>  > cannot be decided, then we support NoPlan with the following additions
>  > to the WebRTC JS API as an option that allows implementing either Plan A
>  > or Plan B  in Javascript.  And even if Plan A vs. Plan B is resolved,
>  > these API additions would still be a big improvement for those WebRTC
>  > applications that don't use SDP for signalling.
>  >
>  > It is a bit long because I have added many comments and examples, but
>  > the actually additions only include two new methods on PeerConnection
>  > and a few new dictionaries.  So don't be overwhelmed :).
>  >
>  >
>  >
>  > Intro: This follows the model of createDataChannel, which has a JS
>  > method on PeerConnection that makes it possible to add data channels
>  > without going through SDP.  Furthermore, just like createDataChannel
>  > allows 2 ways to handle neogitation (the "I know what I'm doing;  Here's
>  > what I want to send; Let me signal everything" mode and the "please take
>  > care of it for me;  send an OPEN message" mode), this also has 2 ways to
>  > handle negotiation (the "I know what I'm doing; Here's what I want to
>  > send; Let me signal everything" mode and the "please take care of it for
>  > me;  send SDP back and forth" mode).
>  >
>  > Following the success of createDataChannel, this allows simple
>  > applications to Just Work and more advanced applications to easily
>  > control what they need to.  In particular, it's possible to use this API
>  > to implement either Plan A or Plan B.
>  >
>  > // The following two method are added to RTCPeerConnection
>  > partial interface RTCPeerConnection {
>  >   // Create a stream that is used to send a source stream.
>  >   // The MediaSendStream.description can be used for signalling.
>  >   // No media is sent until addStream(MediaSendStream) is called.
>  >   LocalMediaStream createLocalStream(MediaStream sourceStream);
>  >
>  >   // Create a stream that is used to receive media from the remote side,
>  >   // given the parameters signalled from MedaiSendStream.description.
>  >   MediaStream createRemoteStream(MediaStreamDescription description);
>  > }____
>
> what would happen if the application adds or removes a track from
> sourceStream (I'm thinking on how what would impact the created
> LocalMediaStream and the MediaStream at the remote end)?____
>
> __ __
>
> __ __
>
> Good question.  I'd have to think about it some more, but off the bat,
> I'd say that on the sender side, adding or removing tracks from the
> source MediaStream would have no effect on what is sent.  In other
> words, it's as if the source MediaStream were cloned when
> createLocalStream was called.  If you want to modify the LocalStream,
> you have to go through the .description.   On the receiver side, it
> would look like any other received MediaStream: if you remove the track,
> it just doesn't play out anymore.____
>
> __ __
>
> ____
>
>      >
>      >
>      > interface LocalMediaStream implements MediaStream {
>      >    // This can be changed at any time, but especially before calling
>      >    // PeerConnection.addStream
>      >    attribute MediaStreamDescription description;
>      > }
>      >
>      >
>      > // Represents the parameters used to either send or receive a stream
>      > // over a PeerConnection.
>      > dictionary MediaStreamDescription {
>      >    MediaStreamTrackDescription[] tracks;
>      > }
>      >
>      >
>      > // Represents the parameters used to either send or receive a
>     track over
>      > // a PeerConnection.  A track has many "flows", which can be grouped
>      > // together.
>      > dictionary MediaStreamTrackDescription {
>      >    // Same as the MediaStreamTrack.id
>      >    DOMString id;
>      >
>      >    // Same as the MediaStreamTrack.kind
>      >    DOMString kind;
>      >
>      >    // A track can have many "flows", such as for Simulcast, FEC, etc.
>      >    // And they can be grouped in arbitrary ways.
>      >    MediaFlowDescription[] flows;
>      >    MediaFlowGroup[] flowGroups;
>      > }
>      >
>      > // Represents the parameters used to either send or receive a "flow"
>      > // over a PeerConnection.  A "flow" is a media that arrives with a
>      > // single, unique SSRC.  One to many flows together make up the media
>      > // for a track.  For example, there may be Simulcast, FEC, and RTX
>      > // flows.
>      > dictionay MediaFlowDescription {
>      >    // The "flow id" must be unique to the track, but need not be
>     unique
>      >    // outside of the track (two tracks could both have a flow
>     with the
>      >    // same flow ID).
>      >    DOMString id;
>      >
>      >    // Each flow can go over its own transport.  If the JS sets
>     this to a
>      >    // transportId that doesn't have a transport setup already, the
>      >    // browser will use SDP negotiation to setup a transport to
>     back that
>      >    // transportId.  If This is set to an MID in the SDP, then
>     that MID's
>      >    // transport is used.
>      >    DOMString transportId;____
>
>     Where from do you get transportId (e.g. if you want to re-use one)?____
>
> __ __
>
> First, the browser would fill it in via createLocalStream.  So in almost
> all cases, the JS would not need to set it.  In very advanced
> applications that want to do tricky things with transports, then the
> value should be the MID for that transport found in the SDP, as it says
> in the comment. ____
>
> ____
>
>
>      >
>      >    // The SSRC used to send the flow.
>      >    unsigned int ssrc;
>      >
>      >    // When used as receive parameters, this indicates the
>     possible list
>      >    // of codecs that might come in for this flow.  For exmample,
>     a given
>      >    // receive flow could be setup to receive any of OPUS, ISAC,
>     or PCMU.
>      >    // When used as send parameters, this indicates that the first
>     codec
>      >    // should be used, but the browser can use send other codecs if it
>      >    // needs to because of either bandwidth or CPU constraints.
>      >    MediaCodecDescription[] codecs;
>      > }
>      >
>      >
>      > dictionary MediaFlowGroup {
>      >    DOMString type;  // "SIM" for Simulcast, "FEC" for FEC, etc
>      >    DOMString[] flowids;
>      > }
>      >
>      > dictionary MediaCodecDescription {
>      >    unsigned byte payloadType;
>      >    DOMString name;
>      >    unsigned int? clockRate;
>      >    unsigned int? bitRate;
>      >    // A grab bag of other fmtp that will need to be further defined.
>      >    MediaCodecParam[] params;
>      > }
>      >
>      > dictionary MediaCodecParam {
>      >    DOMString key;
>      >    DOMString value;
>      > }
>      >
>      >
>      > Notes:
>      >
>      > - When LocalMediaStreams are added using addStream,
>     onnegotiatedneeded
>      > is not called, and those streams are never reflected in future SDP
>      > exchanges.  Indeed, it would be impossible to put them in the SDP
>      > without first resolving if that would be Plan A SDP or Plan B SDP.
>      >
>      > - Just like piles of attributes would need to be defined for Plan
>     A and
>      > for Plan B, similar attributes would need to be defined here
>     (Luckily,
>      >   much work has already been done figuring out what those
>     parameters are :).
>      >
>      >
>      > Pros:
>      >
>      > - Either Plan A or Plan B or could be implemented in Javascript using
>      > this API
>      > - It exposes all the same functionality to the Javascript as SDP,
>     but in
>      > a much nicer format that is much easier to work with.
>      > - Any other signalling mechanism, such as Jingle or CLUE could be
>      > implemented using this API.
>      > - There is almost no risk of signalling glare.
>      > - Debugging errors with misconfigured descriptions should be much
>     easier
>      > with this than with large SDP blobs.
>      >
>      >
>      > Cons:
>      >
>      > - Now there are two slightly different ways to add streams: by
>     creating
>      > a LocalMediaStream first, and not.  This is, however, analogous to
>      > setting "negotiated: true" in createDataChannel.  On way is "Just
>     Work",
>      > and the other is more advanced control.
>      >
>      > - All the options in MediaCodecDescription are a bit complicated.
>      >   Really, this is only necessary because Plan A requires being
>     able to
>      > specify codec parameters per SSRC, and set each flow on different
>      > transports.  If we did not have this requirement, we could simplify.
>      >
>      >
>      > Example Usage:
>      >
>      > // Imagine I have MyApp, handles creating a PeerConnection,
>      > // signalling, and rendering streams.  This is how the new API
>     could be
>      > // used.
>      > var peerConnection = MyApp.createPeerConnection();
>      >
>      > // On sender side:
>      > var stream = MyApp.getMediaStream();
>      > var localStream = peerConnection.createSendStream(stream);
>      > sendStream.description =
>     MyApp.modifyStream(localStream.description)____
>
>     Should it say "localStream.description"?____
>
> __ __
>
> Yes. There was a last-minute rename and I missed a spot.  Anywhere you
> see "sendStream", it should be "localStream".____
>
> ____
>
>
>      > MyApp.signalAddStream(localStream.description, function(response)) {
>      >    if (!response.rejected) {
>      >      // Media will not be sent.
>      >      peerConnection.addStream(localStream);
>      >    }
>      > }
>      >
>      > // On receiver side:
>      > MyApp.onAddStreamSignalled = function(streamDescription) {
>      >    var stream =
>     peerConnection.createReceiveStream(streamDescription);
>      >    MyApp.renderStream(stream);
>      > }
>      >
>      >
>      > // In this exchange, the MediaStreamDescription signalled from the
>      > // sender to the receiver may have looked something like this:
>      >
>      > {
>      >    tracks: [
>      >    {
>      >      id: "audio1",
>      >      kind: "audio",
>      >      flows: [
>      >      {
>      > id: "main",
>      >        transportId: "transport1",
>      >        ssrc: 1111,
>      >        codecs: [
>      >        {
>      >          payloadType: 111,
>      >          name: "opus",
>      >          // ... more codec details
>      >        },
>      >        {
>      >          payloadType: 112,
>      >          name: "pcmu",
>      > // ... more codec details
>      >        }]
>      >     }]
>      >   },
>      >   {
>      >      id: "video1",
>      >      kind: "video",
>      >      flows: [
>      >      {
>      >        id: "sim0",
>      >        transportId: "transport2",
>      >        ssrc: 2222,
>      >        codecs: [
>      >        {
>      >          payloadType: 122,
>      >          name: "vp8"
>      > // ... more codec details
>      >        }]
>      >     },
>      >     {
>      >       id: "sim1",
>      >       transportId: "transport2",
>      >       ssrc: 2223,
>      >       codecs: [
>      >       {
>      >         payloadType: 122,
>      >         name: "vp8",
>      > // ... more codec details
>      >       }]
>      >     },
>      >     {
>      >       id: "sim2",
>      >       transportId: "transport2",
>      >       ssrc: 2224,
>      >       codecs: [
>      >       {
>      >         payloadType: 122,
>      >         name: "vp8",
>      > // ... more codec details
>      >       }]
>      >     },
>      >
>      >     {
>      >       id: "sim0fec",
>      >       transportId: "transport2",
>      >       ssrc: 2225,
>      >       codecs: [
>      >       {
>      >         payloadType: 122,
>      >         name: "vp8",
>      >         // ...
>      >       }]
>      >     }],
>      >     flowGroups: [
>      >     {
>      >       semantics: "SIM",
>      >       ssrcs: [2222, 2223, 2224]
>      >     },
>      >     {
>      >       semantics: "FEC",
>      >       ssrcs: [2222, 2225]
>      >     }]
>      >   }]
>      > }
>      >
>      >
>      > Constructive feedback is welcome :).____
>
> __ __
>
>