Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)

Harald Alvestrand <harald@alvestrand.no> Tue, 18 June 2013 08:11 UTC

Return-Path: <harald@alvestrand.no>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8AF8621F9446 for <rtcweb@ietfa.amsl.com>; Tue, 18 Jun 2013 01:11:52 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -110.598
X-Spam-Level:
X-Spam-Status: No, score=-110.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iS66P6eLJ4gv for <rtcweb@ietfa.amsl.com>; Tue, 18 Jun 2013 01:11:47 -0700 (PDT)
Received: from eikenes.alvestrand.no (eikenes.alvestrand.no [158.38.152.233]) by ietfa.amsl.com (Postfix) with ESMTP id 3DC8221F9944 for <rtcweb@ietf.org>; Tue, 18 Jun 2013 01:11:46 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by eikenes.alvestrand.no (Postfix) with ESMTP id 3D1B139E057 for <rtcweb@ietf.org>; Tue, 18 Jun 2013 10:11:44 +0200 (CEST)
X-Virus-Scanned: Debian amavisd-new at eikenes.alvestrand.no
Received: from eikenes.alvestrand.no ([127.0.0.1]) by localhost (eikenes.alvestrand.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ernyNKj0GbTM for <rtcweb@ietf.org>; Tue, 18 Jun 2013 10:11:38 +0200 (CEST)
Received: from hta-dell.lul.corp.google.com (unknown [74.125.57.89]) by eikenes.alvestrand.no (Postfix) with ESMTPSA id 569F539E03B for <rtcweb@ietf.org>; Tue, 18 Jun 2013 10:11:38 +0200 (CEST)
Message-ID: <51C01639.5050501@alvestrand.no>
Date: Tue, 18 Jun 2013 10:11:37 +0200
From: Harald Alvestrand <harald@alvestrand.no>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130510 Thunderbird/17.0.6
MIME-Version: 1.0
To: rtcweb@ietf.org
References: <CAJrXDUHdoxLTsofiwLBdwBNnCCkCBgjSdbmLaXrNEPODMrsSVA@mail.gmail.com> <1447FA0C20ED5147A1AA0EF02890A64B1C2FC071@ESESSMB209.ericsson.se> <CAJrXDUEL1ynci_bwLydYgvWD=+FQKhurzV3vC0X4LrgBjUMrkA@mail.gmail.com> <57A15FAF9E58F841B2B1651FFE16D28104C918@GENSJZMBX01.msg.int.genesyslab.com> <CAJrXDUHphQq+=vWRNjuRN1BAcUz4kfUf7DEwCPgBBX-esQh0HQ@mail.gmail.com>
In-Reply-To: <CAJrXDUHphQq+=vWRNjuRN1BAcUz4kfUf7DEwCPgBBX-esQh0HQ@mail.gmail.com>
Content-Type: multipart/alternative; boundary="------------020701000808000504050507"
Subject: Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 18 Jun 2013 08:11:53 -0000

On 06/17/2013 06:20 PM, Peter Thatcher wrote:
>
>
>
> On Mon, Jun 17, 2013 at 8:21 AM, Jim Barnett 
> <Jim.Barnett@genesyslab.com <mailto:Jim.Barnett@genesyslab.com>> wrote:
>
>     What can you modify in a LocalMediaStream by editing its
>     description?  Can you add a video track to it?  That would imply
>     calling gUM, and you'd  have to thread it's success and failure
>     callbacks up to the editing operation.  Or is the track list
>     fixed, so that you can only tweak transport info by editing the
>     LocalMediaStream's description?
>
>
> If the JS calls gUM to get a second MediaStreamTrack (say, for 
> screencast), it could simply call createLocalStream again to add it, 
> in a different LocalMediaStream, and the remote end would get a second 
> MediaStream by calling createRemoteStream.  Why would you want more 
> than one video MediaStreamTrack in MediaStream at the same time?
>
> In general, before you start sending or receiving, you can edit 
> everything side of the MediaStreamTrackDescription:  change the ssrcs, 
> codecs, transport, video resolution to send with, what repair flows to 
> include, whether to use simulcast, etc.  Basically, anything on a 
> track level.

This raises an interesting issue: Anything the user is allowed to set, 
the UA has to assume that the user can get wrong. So we need to have 
error handling for AddLocalStream saying "sorry, this is impossible".

Perennial question: Exception or callback?

>
> But I don't really see a use case where you would need to add tracks 
> to a MediaStreamTrackDescription, other than when parsing signalling 
> and building the MediaStreamTrackDescription to send down into 
> createRemoteStream.  Perhaps if we had a use case, we could define 
> such support.  But otherwise, I'd say it's not allowed.
>
>
> -Jim
>
> *From:*rtcweb-bounces@ietf.org <mailto:rtcweb-bounces@ietf.org> 
> [mailto:rtcweb-bounces@ietf.org <mailto:rtcweb-bounces@ietf.org>] *On 
> Behalf Of *Peter Thatcher
> *Sent:* Monday, June 17, 2013 11:13 AM
> *To:* Stefan Håkansson LK
> *Cc:* <rtcweb@ietf.org <mailto:rtcweb@ietf.org>>
> *Subject:* Re: [rtcweb] Proposal for a JS API for NoPlan (adding 
> multiple sources without encoding them in SDP)
>
> Answers:
>
> On Mon, Jun 17, 2013 at 6:58 AM, Stefan Håkansson LK 
> <stefan.lk.hakansson@ericsson.com 
> <mailto:stefan.lk.hakansson@ericsson.com>> wrote:
>
> A couple of questions (to help me understand) in-line //Stefan
>
>
> On 2013-06-17 14:57, Peter Thatcher wrote:
> > Google is in full support of "Plan B" for encoding multiple media
> > sources in SDP, and would like to see the Plan A vs. Plan B decision
> > resolved soon.  Recently, though, a third option, called "NoPlan" has
> > been proposed, but it lacked the details of what a JS API would look
> > like for NoPlan.  Cullen asked to see such an API proposal, and so I
> > have worked with Emil to make one.  He will be adding it to the NoPlan
> > draft, but I will also include it in this email.
> >
> > Again, Google is in full support of "Plan B".  But if Plan A vs. Plan B
> > cannot be decided, then we support NoPlan with the following additions
> > to the WebRTC JS API as an option that allows implementing either Plan A
> > or Plan B  in Javascript.  And even if Plan A vs. Plan B is resolved,
> > these API additions would still be a big improvement for those WebRTC
> > applications that don't use SDP for signalling.
> >
> > It is a bit long because I have added many comments and examples, but
> > the actually additions only include two new methods on PeerConnection
> > and a few new dictionaries.  So don't be overwhelmed :).
> >
> >
> >
> > Intro: This follows the model of createDataChannel, which has a JS
> > method on PeerConnection that makes it possible to add data channels
> > without going through SDP.  Furthermore, just like createDataChannel
> > allows 2 ways to handle neogitation (the "I know what I'm doing;  Here's
> > what I want to send; Let me signal everything" mode and the "please take
> > care of it for me;  send an OPEN message" mode), this also has 2 ways to
> > handle negotiation (the "I know what I'm doing; Here's what I want to
> > send; Let me signal everything" mode and the "please take care of it for
> > me;  send SDP back and forth" mode).
> >
> > Following the success of createDataChannel, this allows simple
> > applications to Just Work and more advanced applications to easily
> > control what they need to.  In particular, it's possible to use this API
> > to implement either Plan A or Plan B.
> >
> > // The following two method are added to RTCPeerConnection
> > partial interface RTCPeerConnection {
> >   // Create a stream that is used to send a source stream.
> >   // The MediaSendStream.description can be used for signalling.
> >   // No media is sent until addStream(MediaSendStream) is called.
> >   LocalMediaStream createLocalStream(MediaStream sourceStream);
> >
> >   // Create a stream that is used to receive media from the remote side,
> >   // given the parameters signalled from MedaiSendStream.description.
> >   MediaStream createRemoteStream(MediaStreamDescription description);
> > }
>
> what would happen if the application adds or removes a track from
> sourceStream (I'm thinking on how what would impact the created
> LocalMediaStream and the MediaStream at the remote end)?
>
> Good question.  I'd have to think about it some more, but off the bat, 
> I'd say that on the sender side, adding or removing tracks from the 
> source MediaStream would have no effect on what is sent.  In other 
> words, it's as if the source MediaStream were cloned when 
> createLocalStream was called.  If you want to modify the LocalStream, 
> you have to go through the .description. On the receiver side, it 
> would look like any other received MediaStream: if you remove the 
> track, it just doesn't play out anymore.
>
>     >
>     >
>     > interface LocalMediaStream implements MediaStream {
>     >    // This can be changed at any time, but especially before calling
>     >    // PeerConnection.addStream
>     >    attribute MediaStreamDescription description;
>     > }
>     >
>     >
>     > // Represents the parameters used to either send or receive a stream
>     > // over a PeerConnection.
>     > dictionary MediaStreamDescription {
>     >  MediaStreamTrackDescription[] tracks;
>     > }
>     >
>     >
>     > // Represents the parameters used to either send or receive a
>     track over
>     > // a PeerConnection.  A track has many "flows", which can be grouped
>     > // together.
>     > dictionary MediaStreamTrackDescription {
>     >    // Same as the MediaStreamTrack.id
>     >    DOMString id;
>     >
>     >    // Same as the MediaStreamTrack.kind
>     >    DOMString kind;
>     >
>     >    // A track can have many "flows", such as for Simulcast, FEC,
>     etc.
>     >    // And they can be grouped in arbitrary ways.
>     >    MediaFlowDescription[] flows;
>     >    MediaFlowGroup[] flowGroups;
>     > }
>     >
>     > // Represents the parameters used to either send or receive a "flow"
>     > // over a PeerConnection.  A "flow" is a media that arrives with a
>     > // single, unique SSRC.  One to many flows together make up the
>     media
>     > // for a track.  For example, there may be Simulcast, FEC, and RTX
>     > // flows.
>     > dictionay MediaFlowDescription {
>     >    // The "flow id" must be unique to the track, but need not be
>     unique
>     >    // outside of the track (two tracks could both have a flow
>     with the
>     >    // same flow ID).
>     >    DOMString id;
>     >
>     >    // Each flow can go over its own transport.  If the JS sets
>     this to a
>     >    // transportId that doesn't have a transport setup already, the
>     >    // browser will use SDP negotiation to setup a transport to
>     back that
>     >    // transportId.  If This is set to an MID in the SDP, then
>     that MID's
>     >    // transport is used.
>     >    DOMString transportId;
>
>     Where from do you get transportId (e.g. if you want to re-use one)?
>
> First, the browser would fill it in via createLocalStream.  So in 
> almost all cases, the JS would not need to set it.  In very advanced 
> applications that want to do tricky things with transports, then the 
> value should be the MID for that transport found in the SDP, as it 
> says in the comment.
>
>
>     >
>     >    // The SSRC used to send the flow.
>     >    unsigned int ssrc;
>     >
>     >    // When used as receive parameters, this indicates the
>     possible list
>     >    // of codecs that might come in for this flow.  For exmample,
>     a given
>     >    // receive flow could be setup to receive any of OPUS, ISAC,
>     or PCMU.
>     >    // When used as send parameters, this indicates that the
>     first codec
>     >    // should be used, but the browser can use send other codecs
>     if it
>     >    // needs to because of either bandwidth or CPU constraints.
>     >    MediaCodecDescription[] codecs;
>     > }
>     >
>     >
>     > dictionary MediaFlowGroup {
>     >    DOMString type;  // "SIM" for Simulcast, "FEC" for FEC, etc
>     >    DOMString[] flowids;
>     > }
>     >
>     > dictionary MediaCodecDescription {
>     >    unsigned byte payloadType;
>     >    DOMString name;
>     >    unsigned int? clockRate;
>     >    unsigned int? bitRate;
>     >    // A grab bag of other fmtp that will need to be further defined.
>     >    MediaCodecParam[] params;
>     > }
>     >
>     > dictionary MediaCodecParam {
>     >    DOMString key;
>     >    DOMString value;
>     > }
>     >
>     >
>     > Notes:
>     >
>     > - When LocalMediaStreams are added using addStream,
>     onnegotiatedneeded
>     > is not called, and those streams are never reflected in future SDP
>     > exchanges.  Indeed, it would be impossible to put them in the SDP
>     > without first resolving if that would be Plan A SDP or Plan B SDP.
>     >
>     > - Just like piles of attributes would need to be defined for
>     Plan A and
>     > for Plan B, similar attributes would need to be defined here
>     (Luckily,
>     >   much work has already been done figuring out what those
>     parameters are :).
>     >
>     >
>     > Pros:
>     >
>     > - Either Plan A or Plan B or could be implemented in Javascript
>     using
>     > this API
>     > - It exposes all the same functionality to the Javascript as
>     SDP, but in
>     > a much nicer format that is much easier to work with.
>     > - Any other signalling mechanism, such as Jingle or CLUE could be
>     > implemented using this API.
>     > - There is almost no risk of signalling glare.
>     > - Debugging errors with misconfigured descriptions should be
>     much easier
>     > with this than with large SDP blobs.
>     >
>     >
>     > Cons:
>     >
>     > - Now there are two slightly different ways to add streams: by
>     creating
>     > a LocalMediaStream first, and not.  This is, however, analogous to
>     > setting "negotiated: true" in createDataChannel.  On way is
>     "Just Work",
>     > and the other is more advanced control.
>     >
>     > - All the options in MediaCodecDescription are a bit complicated.
>     >   Really, this is only necessary because Plan A requires being
>     able to
>     > specify codec parameters per SSRC, and set each flow on different
>     > transports.  If we did not have this requirement, we could simplify.
>     >
>     >
>     > Example Usage:
>     >
>     > // Imagine I have MyApp, handles creating a PeerConnection,
>     > // signalling, and rendering streams.  This is how the new API
>     could be
>     > // used.
>     > var peerConnection = MyApp.createPeerConnection();
>     >
>     > // On sender side:
>     > var stream = MyApp.getMediaStream();
>     > var localStream = peerConnection.createSendStream(stream);
>     > sendStream.description = MyApp.modifyStream(localStream.description)
>
>     Should it say "localStream.description"?
>
> Yes. There was a last-minute rename and I missed a spot.  Anywhere you 
> see "sendStream", it should be "localStream".
>
>
>     > MyApp.signalAddStream(localStream.description, function(response)) {
>     >    if (!response.rejected) {
>     >      // Media will not be sent.
>     >  peerConnection.addStream(localStream);
>     >    }
>     > }
>     >
>     > // On receiver side:
>     > MyApp.onAddStreamSignalled = function(streamDescription) {
>     >    var stream =
>     peerConnection.createReceiveStream(streamDescription);
>     >    MyApp.renderStream(stream);
>     > }
>     >
>     >
>     > // In this exchange, the MediaStreamDescription signalled from the
>     > // sender to the receiver may have looked something like this:
>     >
>     > {
>     >    tracks: [
>     >    {
>     >      id: "audio1",
>     >      kind: "audio",
>     >      flows: [
>     >      {
>     > id: "main",
>     >        transportId: "transport1",
>     >        ssrc: 1111,
>     >        codecs: [
>     >        {
>     >          payloadType: 111,
>     >          name: "opus",
>     >          // ... more codec details
>     >        },
>     >        {
>     >          payloadType: 112,
>     >          name: "pcmu",
>     > // ... more codec details
>     >        }]
>     >     }]
>     >   },
>     >   {
>     >      id: "video1",
>     >      kind: "video",
>     >      flows: [
>     >      {
>     >        id: "sim0",
>     >        transportId: "transport2",
>     >        ssrc: 2222,
>     >        codecs: [
>     >        {
>     >          payloadType: 122,
>     >          name: "vp8"
>     > // ... more codec details
>     >        }]
>     >     },
>     >     {
>     >       id: "sim1",
>     >       transportId: "transport2",
>     >       ssrc: 2223,
>     >       codecs: [
>     >       {
>     >         payloadType: 122,
>     >         name: "vp8",
>     > // ... more codec details
>     >       }]
>     >     },
>     >     {
>     >       id: "sim2",
>     >       transportId: "transport2",
>     >       ssrc: 2224,
>     >       codecs: [
>     >       {
>     >         payloadType: 122,
>     >         name: "vp8",
>     > // ... more codec details
>     >       }]
>     >     },
>     >
>     >     {
>     >       id: "sim0fec",
>     >       transportId: "transport2",
>     >       ssrc: 2225,
>     >       codecs: [
>     >       {
>     >         payloadType: 122,
>     >         name: "vp8",
>     >         // ...
>     >       }]
>     >     }],
>     >     flowGroups: [
>     >     {
>     >       semantics: "SIM",
>     >       ssrcs: [2222, 2223, 2224]
>     >     },
>     >     {
>     >       semantics: "FEC",
>     >       ssrcs: [2222, 2225]
>     >     }]
>     >   }]
>     > }
>     >
>     >
>     > Constructive feedback is welcome :).
>
>
>
>
> _______________________________________________
> rtcweb mailing list
> rtcweb@ietf.org
> https://www.ietf.org/mailman/listinfo/rtcweb