Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)

Martin Thomson <martin.thomson@gmail.com> Mon, 17 June 2013 18:57 UTC

Return-Path: <martin.thomson@gmail.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6F06C21F9A86 for <rtcweb@ietfa.amsl.com>; Mon, 17 Jun 2013 11:57:47 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.505
X-Spam-Level:
X-Spam-Status: No, score=-2.505 tagged_above=-999 required=5 tests=[AWL=0.095, BAYES_00=-2.599, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2Hvvfg2cVI8R for <rtcweb@ietfa.amsl.com>; Mon, 17 Jun 2013 11:57:46 -0700 (PDT)
Received: from mail-wg0-x234.google.com (mail-wg0-x234.google.com [IPv6:2a00:1450:400c:c00::234]) by ietfa.amsl.com (Postfix) with ESMTP id EC2AB21F9A74 for <rtcweb@ietf.org>; Mon, 17 Jun 2013 11:57:45 -0700 (PDT)
Received: by mail-wg0-f52.google.com with SMTP id b12so2743658wgh.7 for <rtcweb@ietf.org>; Mon, 17 Jun 2013 11:57:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=8MiANTaxlB4K8KPhd2mhm9gA0ES3E/JeT5cxvHiBRg4=; b=r1+MXt+M+cFqoqQkCM+4ekh8ZJNfKXyCI9l9lL7kRrje7BNgS+kCRrwlBtMDQBHY27 Xf9MkFhQvnFeQWDgFBTyHAZQeGNBbw+yTi/EjcS1qIDcrlsM1wKHXjnGXrzcOCUgMEMY vizVLzjDlfXLj4AZwdQurMSAZ0RuYkWtsxQd1tcWVPWxkKdDdr0c8r+vbTXf+MLdASeK 3ztXTYWlp+I1S38aVbABnEEJ/zUcoyRkpChZg757VVPKlos7v2m5N8Fbbe13bYAA06pZ bCenWVszjtWozOWQyObw6Sli7ROBWiVk6x/9uYrEwlpYhZ/nVtuvztdHWFgqE3KC2vbJ aGcQ==
MIME-Version: 1.0
X-Received: by 10.180.39.236 with SMTP id s12mr5664069wik.14.1371495464598; Mon, 17 Jun 2013 11:57:44 -0700 (PDT)
Received: by 10.194.60.46 with HTTP; Mon, 17 Jun 2013 11:57:44 -0700 (PDT)
In-Reply-To: <CAJrXDUHdoxLTsofiwLBdwBNnCCkCBgjSdbmLaXrNEPODMrsSVA@mail.gmail.com>
References: <CAJrXDUHdoxLTsofiwLBdwBNnCCkCBgjSdbmLaXrNEPODMrsSVA@mail.gmail.com>
Date: Mon, 17 Jun 2013 11:57:44 -0700
Message-ID: <CABkgnnUmRpanfpwryyiCUsOdMLzrd74n-4LXaj_AK3aLe0yQ8Q@mail.gmail.com>
From: Martin Thomson <martin.thomson@gmail.com>
To: Peter Thatcher <pthatcher@google.com>
Content-Type: text/plain; charset=UTF-8
Cc: "<rtcweb@ietf.org>" <rtcweb@ietf.org>
Subject: Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 17 Jun 2013 18:57:47 -0000

Maybe you'd expect me to be more supportive of something that looked
so much like CU-RTC-Web.  It inherits all the worst properties of JSEP
(Offer/Answer, SDP editing) with a partial implementation of a clean
API.

It's comment 22-lite.  It's an abomination.  If you are going to do
this, do it properly.

On 17 June 2013 05:57, Peter Thatcher <pthatcher@google.com>; wrote:
> Google is in full support of "Plan B" for encoding multiple media sources in
> SDP, and would like to see the Plan A vs. Plan B decision resolved soon.
> Recently, though, a third option, called "NoPlan" has been proposed, but it
> lacked the details of what a JS API would look like for NoPlan.  Cullen
> asked to see such an API proposal, and so I have worked with Emil to make
> one.  He will be adding it to the NoPlan draft, but I will also include it
> in this email.
>
> Again, Google is in full support of "Plan B".  But if Plan A vs. Plan B
> cannot be decided, then we support NoPlan with the following additions to
> the WebRTC JS API as an option that allows implementing either Plan A or
> Plan B  in Javascript.  And even if Plan A vs. Plan B is resolved, these API
> additions would still be a big improvement for those WebRTC applications
> that don't use SDP for signalling.
>
> It is a bit long because I have added many comments and examples, but the
> actually additions only include two new methods on PeerConnection and a few
> new dictionaries.  So don't be overwhelmed :).
>
>
>
> Intro: This follows the model of createDataChannel, which has a JS method on
> PeerConnection that makes it possible to add data channels without going
> through SDP.  Furthermore, just like createDataChannel allows 2 ways to
> handle neogitation (the "I know what I'm doing;  Here's what I want to send;
> Let me signal everything" mode and the "please take care of it for me;  send
> an OPEN message" mode), this also has 2 ways to handle negotiation (the "I
> know what I'm doing; Here's what I want to send; Let me signal everything"
> mode and the "please take care of it for me;  send SDP back and forth"
> mode).
>
> Following the success of createDataChannel, this allows simple applications
> to Just Work and more advanced applications to easily control what they need
> to.  In particular, it's possible to use this API to implement either Plan A
> or Plan B.
>
> // The following two method are added to RTCPeerConnection
> partial interface RTCPeerConnection {
>  // Create a stream that is used to send a source stream.
>  // The MediaSendStream.description can be used for signalling.
>  // No media is sent until addStream(MediaSendStream) is called.
>  LocalMediaStream createLocalStream(MediaStream sourceStream);
>
>  // Create a stream that is used to receive media from the remote side,
>  // given the parameters signalled from MedaiSendStream.description.
>  MediaStream createRemoteStream(MediaStreamDescription description);
> }
>
>
> interface LocalMediaStream implements MediaStream {
>   // This can be changed at any time, but especially before calling
>   // PeerConnection.addStream
>   attribute MediaStreamDescription description;
> }
>
>
> // Represents the parameters used to either send or receive a stream
> // over a PeerConnection.
> dictionary MediaStreamDescription {
>   MediaStreamTrackDescription[] tracks;
> }
>
>
> // Represents the parameters used to either send or receive a track over //
> a PeerConnection.  A track has many "flows", which can be grouped
> // together.
> dictionary MediaStreamTrackDescription {
>   // Same as the MediaStreamTrack.id
>   DOMString id;
>
>   // Same as the MediaStreamTrack.kind
>   DOMString kind;
>
>   // A track can have many "flows", such as for Simulcast, FEC, etc.
>   // And they can be grouped in arbitrary ways.
>   MediaFlowDescription[] flows;
>   MediaFlowGroup[] flowGroups;
> }
>
> // Represents the parameters used to either send or receive a "flow"
> // over a PeerConnection.  A "flow" is a media that arrives with a
> // single, unique SSRC.  One to many flows together make up the media
> // for a track.  For example, there may be Simulcast, FEC, and RTX
> // flows.
> dictionay MediaFlowDescription {
>   // The "flow id" must be unique to the track, but need not be unique
>   // outside of the track (two tracks could both have a flow with the
>   // same flow ID).
>   DOMString id;
>
>   // Each flow can go over its own transport.  If the JS sets this to a
>   // transportId that doesn't have a transport setup already, the
>   // browser will use SDP negotiation to setup a transport to back that
>   // transportId.  If This is set to an MID in the SDP, then that MID's
>   // transport is used.
>   DOMString transportId;
>
>   // The SSRC used to send the flow.
>   unsigned int ssrc;
>
>   // When used as receive parameters, this indicates the possible list
>   // of codecs that might come in for this flow.  For exmample, a given
>   // receive flow could be setup to receive any of OPUS, ISAC, or PCMU.
>   // When used as send parameters, this indicates that the first codec
>   // should be used, but the browser can use send other codecs if it
>   // needs to because of either bandwidth or CPU constraints.
>   MediaCodecDescription[] codecs;
> }
>
>
> dictionary MediaFlowGroup {
>   DOMString type;  // "SIM" for Simulcast, "FEC" for FEC, etc
>   DOMString[] flowids;
> }
>
> dictionary MediaCodecDescription {
>   unsigned byte payloadType;
>   DOMString name;
>   unsigned int? clockRate;
>   unsigned int? bitRate;
>   // A grab bag of other fmtp that will need to be further defined.
>   MediaCodecParam[] params;
> }
>
> dictionary MediaCodecParam {
>   DOMString key;
>   DOMString value;
> }
>
>
> Notes:
>
> - When LocalMediaStreams are added using addStream, onnegotiatedneeded is
> not called, and those streams are never reflected in future SDP exchanges.
> Indeed, it would be impossible to put them in the SDP without first
> resolving if that would be Plan A SDP or Plan B SDP.
>
> - Just like piles of attributes would need to be defined for Plan A and for
> Plan B, similar attributes would need to be defined here (Luckily,  much
> work has already been done figuring out what those parameters are :).
>
>
> Pros:
>
> - Either Plan A or Plan B or could be implemented in Javascript using this
> API
> - It exposes all the same functionality to the Javascript as SDP, but in a
> much nicer format that is much easier to work with.
> - Any other signalling mechanism, such as Jingle or CLUE could be
> implemented using this API.
> - There is almost no risk of signalling glare.
> - Debugging errors with misconfigured descriptions should be much easier
> with this than with large SDP blobs.
>
>
> Cons:
>
> - Now there are two slightly different ways to add streams: by creating a
> LocalMediaStream first, and not.  This is, however, analogous to setting
> "negotiated: true" in createDataChannel.  On way is "Just Work", and the
> other is more advanced control.
>
> - All the options in MediaCodecDescription are a bit complicated.  Really,
> this is only necessary because Plan A requires being able to specify codec
> parameters per SSRC, and set each flow on different transports.  If we did
> not have this requirement, we could simplify.
>
>
> Example Usage:
>
> // Imagine I have MyApp, handles creating a PeerConnection,
> // signalling, and rendering streams.  This is how the new API could be
> // used.
> var peerConnection = MyApp.createPeerConnection();
>
> // On sender side:
> var stream = MyApp.getMediaStream();
> var localStream = peerConnection.createSendStream(stream);
> sendStream.description = MyApp.modifyStream(localStream.description)
> MyApp.signalAddStream(localStream.description, function(response)) {
>   if (!response.rejected) {
>     // Media will not be sent.
>     peerConnection.addStream(localStream);
>   }
> }
>
> // On receiver side:
> MyApp.onAddStreamSignalled = function(streamDescription) {
>   var stream = peerConnection.createReceiveStream(streamDescription);
>   MyApp.renderStream(stream);
> }
>
>
> // In this exchange, the MediaStreamDescription signalled from the
> // sender to the receiver may have looked something like this:
>
> {
>   tracks: [
>   {
>     id: "audio1",
>     kind: "audio",
>     flows: [
>     {
>       id: "main",
>       transportId: "transport1",
>       ssrc: 1111,
>       codecs: [
>       {
>         payloadType: 111,
>         name: "opus",
>         // ... more codec details
>       },
>       {
>         payloadType: 112,
>         name: "pcmu",
>         // ... more codec details
>       }]
>    }]
>  },
>  {
>     id: "video1",
>     kind: "video",
>     flows: [
>     {
>       id: "sim0",
>       transportId: "transport2",
>       ssrc: 2222,
>       codecs: [
>       {
>         payloadType: 122,
>         name: "vp8"
>         // ... more codec details
>       }]
>    },
>    {
>      id: "sim1",
>      transportId: "transport2",
>      ssrc: 2223,
>      codecs: [
>      {
>        payloadType: 122,
>        name: "vp8",
>        // ... more codec details
>      }]
>    },
>    {
>      id: "sim2",
>      transportId: "transport2",
>      ssrc: 2224,
>      codecs: [
>      {
>        payloadType: 122,
>        name: "vp8",
>        // ... more codec details
>      }]
>    },
>
>    {
>      id: "sim0fec",
>      transportId: "transport2",
>      ssrc: 2225,
>      codecs: [
>      {
>        payloadType: 122,
>        name: "vp8",
>        // ...
>      }]
>    }],
>    flowGroups: [
>    {
>      semantics: "SIM",
>      ssrcs: [2222, 2223, 2224]
>    },
>    {
>      semantics: "FEC",
>      ssrcs: [2222, 2225]
>    }]
>  }]
> }
>
>
> Constructive feedback is welcome :).
>
> _______________________________________________
> rtcweb mailing list
> rtcweb@ietf.org
> https://www.ietf.org/mailman/listinfo/rtcweb
>