Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)

Peter Thatcher <pthatcher@google.com> Mon, 17 June 2013 19:26 UTC

Return-Path: <pthatcher@google.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 867CD21F9D24 for <rtcweb@ietfa.amsl.com>; Mon, 17 Jun 2013 12:26:49 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.827
X-Spam-Level:
X-Spam-Status: No, score=-1.827 tagged_above=-999 required=5 tests=[AWL=0.150, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jOqopmaq8S7Q for <rtcweb@ietfa.amsl.com>; Mon, 17 Jun 2013 12:26:48 -0700 (PDT)
Received: from mail-pb0-x234.google.com (mail-pb0-x234.google.com [IPv6:2607:f8b0:400e:c01::234]) by ietfa.amsl.com (Postfix) with ESMTP id DBC0B21F8FF3 for <rtcweb@ietf.org>; Mon, 17 Jun 2013 12:26:47 -0700 (PDT)
Received: by mail-pb0-f52.google.com with SMTP id xa12so3087971pbc.39 for <rtcweb@ietf.org>; Mon, 17 Jun 2013 12:26:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=X8qotrMCpY/s52kZRy25zxvEk+Qx/K7jlTyE78DKXmM=; b=LkBEjfkpi8o23j7cDDcimigIrMsjdQeWyWpB0YEDYqU6WJnlt9sI1DnKYo30T+C261 7zcha0qdyCxhwVVZG52UhCg8XapK4bhLP1JFOatY5nUQVPjSEmA133hTR4wMqZtwO+HM SCaaSosGKAuUgLdqEFK2B3ZyeReXts1nvtbR0WzTO2xsMvn9HmJfMvQwnC+0FkI9kZRk DXPnGll7/9VS3+bwZx4hwazUKEV+yfv/jfdd2mMkQfdgMhBIeY7Mv4/CRzidhc1HpUdO iCfIFzuWCmcRI4gcFranhHHHMG6flNg5F3L618tdP3ruTyfbh2km/lAdOvaRGLaC+xDy zjpA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:x-gm-message-state; bh=X8qotrMCpY/s52kZRy25zxvEk+Qx/K7jlTyE78DKXmM=; b=XvHSzSCYGY7mH9XGppOejtW/uACtYphSA2r26/bNCVA4/KgMqqlJt/DIPBS3VqQt8w gLpowPl/KVrFJkd51QItL2KCgNg0/c7NrO8CYWPxsLi8lFLVh+Lfdm1bigPBC7tWxPsr Xcw2KScdPoIbJSASfJ240tblHmLCVZy7aM4cxpgGAYtKK2RslLdj7d2ecIZPWSprANP3 NL5YU3BAEL9gbUe7JQA5HvwnsOToW2FMlk5GhzWDAwFsfRgENv2/U4YYjFq8ZMUTa5Yz T2T5b2LLlXkOGQfriYRDARuRh84U5jM5MSl2ct9onSZMEKgBYDIYy6NVvJPtC4dCvLKV czCQ==
X-Received: by 10.68.1.226 with SMTP id 2mr14386627pbp.150.1371497207535; Mon, 17 Jun 2013 12:26:47 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.66.88.8 with HTTP; Mon, 17 Jun 2013 12:26:07 -0700 (PDT)
In-Reply-To: <CABkgnnUmRpanfpwryyiCUsOdMLzrd74n-4LXaj_AK3aLe0yQ8Q@mail.gmail.com>
References: <CAJrXDUHdoxLTsofiwLBdwBNnCCkCBgjSdbmLaXrNEPODMrsSVA@mail.gmail.com> <CABkgnnUmRpanfpwryyiCUsOdMLzrd74n-4LXaj_AK3aLe0yQ8Q@mail.gmail.com>
From: Peter Thatcher <pthatcher@google.com>
Date: Mon, 17 Jun 2013 12:26:07 -0700
Message-ID: <CAJrXDUGnEwtsGZwUUqQgH0vDnMPy=XxqwQB9fpNcW9yQDhFt4w@mail.gmail.com>
To: Martin Thomson <martin.thomson@gmail.com>
Content-Type: multipart/alternative; boundary=bcaec5314817ddbb9004df5e933c
X-Gm-Message-State: ALoCoQkh4PE0aGQiPT6ibbOjGhGEeFCwdsPde8RRbg+S3dhgNe+2R0dFi/HHphNXJLXSpzx/q3LEa+0n5Mu8xQoVxvp0zeM9X7l9/+zD8MXtt/A/5ECQ+mnYCJKGSzj+D30dv5rpaXkLZ+Zn0dprVpZtwBLjhROdjHDOBBTXo6/PWrWYZTTpEf0htaQ87lP/0M+J9N2BqEUB
Cc: "<rtcweb@ietf.org>" <rtcweb@ietf.org>
Subject: Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 17 Jun 2013 19:26:49 -0000

Yes, I was expecting you to be more supportive.  I'm surprised out how your
want "all or nothing".  I'm afraid if our options for a clean API are all
or nothing, we'll just end up with nothing.  I'd prefer to try incremental
improvements to word toward (eventually) a clean API.

Do you think it is impossible to work toward a clean API in an incremental
approach?  If you think it's possible, I'd like to hear your thoughts on
how.


By the way, these API additions would greatly minimize the amount of SDP
editing necessary for JS clients that don't use SDP for signalling.  And
later incremental improvements could reduce it further.  Also, it's no
longer necessary to do offer/answer for adding tracks.  It's only the
intial PeerConnection setup that needs to do Offer/Answer.  So, it doesn't
inherit all the problems quite as much as you described.  It may be
slightly abominable, but I certainly consider it less abominable than the
SDP editing necessary without it.


On Mon, Jun 17, 2013 at 11:57 AM, Martin Thomson
<martin.thomson@gmail.com>wrote:

> Maybe you'd expect me to be more supportive of something that looked
> so much like CU-RTC-Web.  It inherits all the worst properties of JSEP
> (Offer/Answer, SDP editing) with a partial implementation of a clean
> API.
>
> It's comment 22-lite.  It's an abomination.  If you are going to do
> this, do it properly.
>
> On 17 June 2013 05:57, Peter Thatcher <pthatcher@google.com>; wrote:
> > Google is in full support of "Plan B" for encoding multiple media
> sources in
> > SDP, and would like to see the Plan A vs. Plan B decision resolved soon.
> > Recently, though, a third option, called "NoPlan" has been proposed, but
> it
> > lacked the details of what a JS API would look like for NoPlan.  Cullen
> > asked to see such an API proposal, and so I have worked with Emil to make
> > one.  He will be adding it to the NoPlan draft, but I will also include
> it
> > in this email.
> >
> > Again, Google is in full support of "Plan B".  But if Plan A vs. Plan B
> > cannot be decided, then we support NoPlan with the following additions to
> > the WebRTC JS API as an option that allows implementing either Plan A or
> > Plan B  in Javascript.  And even if Plan A vs. Plan B is resolved, these
> API
> > additions would still be a big improvement for those WebRTC applications
> > that don't use SDP for signalling.
> >
> > It is a bit long because I have added many comments and examples, but the
> > actually additions only include two new methods on PeerConnection and a
> few
> > new dictionaries.  So don't be overwhelmed :).
> >
> >
> >
> > Intro: This follows the model of createDataChannel, which has a JS
> method on
> > PeerConnection that makes it possible to add data channels without going
> > through SDP.  Furthermore, just like createDataChannel allows 2 ways to
> > handle neogitation (the "I know what I'm doing;  Here's what I want to
> send;
> > Let me signal everything" mode and the "please take care of it for me;
>  send
> > an OPEN message" mode), this also has 2 ways to handle negotiation (the
> "I
> > know what I'm doing; Here's what I want to send; Let me signal
> everything"
> > mode and the "please take care of it for me;  send SDP back and forth"
> > mode).
> >
> > Following the success of createDataChannel, this allows simple
> applications
> > to Just Work and more advanced applications to easily control what they
> need
> > to.  In particular, it's possible to use this API to implement either
> Plan A
> > or Plan B.
> >
> > // The following two method are added to RTCPeerConnection
> > partial interface RTCPeerConnection {
> >  // Create a stream that is used to send a source stream.
> >  // The MediaSendStream.description can be used for signalling.
> >  // No media is sent until addStream(MediaSendStream) is called.
> >  LocalMediaStream createLocalStream(MediaStream sourceStream);
> >
> >  // Create a stream that is used to receive media from the remote side,
> >  // given the parameters signalled from MedaiSendStream.description.
> >  MediaStream createRemoteStream(MediaStreamDescription description);
> > }
> >
> >
> > interface LocalMediaStream implements MediaStream {
> >   // This can be changed at any time, but especially before calling
> >   // PeerConnection.addStream
> >   attribute MediaStreamDescription description;
> > }
> >
> >
> > // Represents the parameters used to either send or receive a stream
> > // over a PeerConnection.
> > dictionary MediaStreamDescription {
> >   MediaStreamTrackDescription[] tracks;
> > }
> >
> >
> > // Represents the parameters used to either send or receive a track over
> //
> > a PeerConnection.  A track has many "flows", which can be grouped
> > // together.
> > dictionary MediaStreamTrackDescription {
> >   // Same as the MediaStreamTrack.id
> >   DOMString id;
> >
> >   // Same as the MediaStreamTrack.kind
> >   DOMString kind;
> >
> >   // A track can have many "flows", such as for Simulcast, FEC, etc.
> >   // And they can be grouped in arbitrary ways.
> >   MediaFlowDescription[] flows;
> >   MediaFlowGroup[] flowGroups;
> > }
> >
> > // Represents the parameters used to either send or receive a "flow"
> > // over a PeerConnection.  A "flow" is a media that arrives with a
> > // single, unique SSRC.  One to many flows together make up the media
> > // for a track.  For example, there may be Simulcast, FEC, and RTX
> > // flows.
> > dictionay MediaFlowDescription {
> >   // The "flow id" must be unique to the track, but need not be unique
> >   // outside of the track (two tracks could both have a flow with the
> >   // same flow ID).
> >   DOMString id;
> >
> >   // Each flow can go over its own transport.  If the JS sets this to a
> >   // transportId that doesn't have a transport setup already, the
> >   // browser will use SDP negotiation to setup a transport to back that
> >   // transportId.  If This is set to an MID in the SDP, then that MID's
> >   // transport is used.
> >   DOMString transportId;
> >
> >   // The SSRC used to send the flow.
> >   unsigned int ssrc;
> >
> >   // When used as receive parameters, this indicates the possible list
> >   // of codecs that might come in for this flow.  For exmample, a given
> >   // receive flow could be setup to receive any of OPUS, ISAC, or PCMU.
> >   // When used as send parameters, this indicates that the first codec
> >   // should be used, but the browser can use send other codecs if it
> >   // needs to because of either bandwidth or CPU constraints.
> >   MediaCodecDescription[] codecs;
> > }
> >
> >
> > dictionary MediaFlowGroup {
> >   DOMString type;  // "SIM" for Simulcast, "FEC" for FEC, etc
> >   DOMString[] flowids;
> > }
> >
> > dictionary MediaCodecDescription {
> >   unsigned byte payloadType;
> >   DOMString name;
> >   unsigned int? clockRate;
> >   unsigned int? bitRate;
> >   // A grab bag of other fmtp that will need to be further defined.
> >   MediaCodecParam[] params;
> > }
> >
> > dictionary MediaCodecParam {
> >   DOMString key;
> >   DOMString value;
> > }
> >
> >
> > Notes:
> >
> > - When LocalMediaStreams are added using addStream, onnegotiatedneeded is
> > not called, and those streams are never reflected in future SDP
> exchanges.
> > Indeed, it would be impossible to put them in the SDP without first
> > resolving if that would be Plan A SDP or Plan B SDP.
> >
> > - Just like piles of attributes would need to be defined for Plan A and
> for
> > Plan B, similar attributes would need to be defined here (Luckily,  much
> > work has already been done figuring out what those parameters are :).
> >
> >
> > Pros:
> >
> > - Either Plan A or Plan B or could be implemented in Javascript using
> this
> > API
> > - It exposes all the same functionality to the Javascript as SDP, but in
> a
> > much nicer format that is much easier to work with.
> > - Any other signalling mechanism, such as Jingle or CLUE could be
> > implemented using this API.
> > - There is almost no risk of signalling glare.
> > - Debugging errors with misconfigured descriptions should be much easier
> > with this than with large SDP blobs.
> >
> >
> > Cons:
> >
> > - Now there are two slightly different ways to add streams: by creating a
> > LocalMediaStream first, and not.  This is, however, analogous to setting
> > "negotiated: true" in createDataChannel.  On way is "Just Work", and the
> > other is more advanced control.
> >
> > - All the options in MediaCodecDescription are a bit complicated.
>  Really,
> > this is only necessary because Plan A requires being able to specify
> codec
> > parameters per SSRC, and set each flow on different transports.  If we
> did
> > not have this requirement, we could simplify.
> >
> >
> > Example Usage:
> >
> > // Imagine I have MyApp, handles creating a PeerConnection,
> > // signalling, and rendering streams.  This is how the new API could be
> > // used.
> > var peerConnection = MyApp.createPeerConnection();
> >
> > // On sender side:
> > var stream = MyApp.getMediaStream();
> > var localStream = peerConnection.createSendStream(stream);
> > sendStream.description = MyApp.modifyStream(localStream.description)
> > MyApp.signalAddStream(localStream.description, function(response)) {
> >   if (!response.rejected) {
> >     // Media will not be sent.
> >     peerConnection.addStream(localStream);
> >   }
> > }
> >
> > // On receiver side:
> > MyApp.onAddStreamSignalled = function(streamDescription) {
> >   var stream = peerConnection.createReceiveStream(streamDescription);
> >   MyApp.renderStream(stream);
> > }
> >
> >
> > // In this exchange, the MediaStreamDescription signalled from the
> > // sender to the receiver may have looked something like this:
> >
> > {
> >   tracks: [
> >   {
> >     id: "audio1",
> >     kind: "audio",
> >     flows: [
> >     {
> >       id: "main",
> >       transportId: "transport1",
> >       ssrc: 1111,
> >       codecs: [
> >       {
> >         payloadType: 111,
> >         name: "opus",
> >         // ... more codec details
> >       },
> >       {
> >         payloadType: 112,
> >         name: "pcmu",
> >         // ... more codec details
> >       }]
> >    }]
> >  },
> >  {
> >     id: "video1",
> >     kind: "video",
> >     flows: [
> >     {
> >       id: "sim0",
> >       transportId: "transport2",
> >       ssrc: 2222,
> >       codecs: [
> >       {
> >         payloadType: 122,
> >         name: "vp8"
> >         // ... more codec details
> >       }]
> >    },
> >    {
> >      id: "sim1",
> >      transportId: "transport2",
> >      ssrc: 2223,
> >      codecs: [
> >      {
> >        payloadType: 122,
> >        name: "vp8",
> >        // ... more codec details
> >      }]
> >    },
> >    {
> >      id: "sim2",
> >      transportId: "transport2",
> >      ssrc: 2224,
> >      codecs: [
> >      {
> >        payloadType: 122,
> >        name: "vp8",
> >        // ... more codec details
> >      }]
> >    },
> >
> >    {
> >      id: "sim0fec",
> >      transportId: "transport2",
> >      ssrc: 2225,
> >      codecs: [
> >      {
> >        payloadType: 122,
> >        name: "vp8",
> >        // ...
> >      }]
> >    }],
> >    flowGroups: [
> >    {
> >      semantics: "SIM",
> >      ssrcs: [2222, 2223, 2224]
> >    },
> >    {
> >      semantics: "FEC",
> >      ssrcs: [2222, 2225]
> >    }]
> >  }]
> > }
> >
> >
> > Constructive feedback is welcome :).
> >
> > _______________________________________________
> > rtcweb mailing list
> > rtcweb@ietf.org
> > https://www.ietf.org/mailman/listinfo/rtcweb
> >
>