Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)

Emil Ivov <emcho@jitsi.org> Mon, 17 June 2013 19:10 UTC

Return-Path: <emil@sip-communicator.org>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 376BC21F9057 for <rtcweb@ietfa.amsl.com>; Mon, 17 Jun 2013 12:10:01 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.562
X-Spam-Level:
X-Spam-Status: No, score=-2.562 tagged_above=-999 required=5 tests=[AWL=0.038, BAYES_00=-2.599, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SqE5CQdgKww4 for <rtcweb@ietfa.amsl.com>; Mon, 17 Jun 2013 12:09:59 -0700 (PDT)
Received: from mail-we0-x232.google.com (mail-we0-x232.google.com [IPv6:2a00:1450:400c:c03::232]) by ietfa.amsl.com (Postfix) with ESMTP id 7858921F9A76 for <rtcweb@ietf.org>; Mon, 17 Jun 2013 12:09:59 -0700 (PDT)
Received: by mail-we0-f178.google.com with SMTP id u53so2724477wes.9 for <rtcweb@ietf.org>; Mon, 17 Jun 2013 12:09:55 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:organization:user-agent:mime-version:to:cc :subject:references:in-reply-to:content-type :content-transfer-encoding:x-gm-message-state; bh=wwQIJjcp0hg3Q5Y5mxa6v0FBD9GzV3F6V0b9CUkoDy8=; b=JvTDFuTrra0fWN46s3nS9hHlzL2eoV4WYkYxuMQxv49r3SPA7rHBpU9+Kd0b8GxgsJ hzANgdJO3shhzkIwPcbrOp3gXTyyrljSWXkUxhVFzC136VpJAwTzAupxDWGyec6JgwmR KdEJNEkn0AwNOMgp+K1sY6WAD7Yo29I5iA4eByi9JY6V14PWoG9GQkqZTRbNURK6l8LZ UO1XFPUFazNr9KGbmBT16ro1GVpzo3rcKTAZnfMd21TbzUeH3bICel7M5qRUZ4s1qiUo rQ0N9/deRsi13ep7l3dhvaEk4r8L2Xwumd8U2eqTQqpZcVTOnJvCI01FEy2IneIP4ms0 NZGA==
X-Received: by 10.180.11.166 with SMTP id r6mr5618839wib.45.1371496195802; Mon, 17 Jun 2013 12:09:55 -0700 (PDT)
Received: from camionet.local ([2a01:e35:2e2c:f600:b43d:dd37:b9b1:450]) by mx.google.com with ESMTPSA id r9sm24132931wik.1.2013.06.17.12.09.53 for <multiple recipients> (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 17 Jun 2013 12:09:54 -0700 (PDT)
Message-ID: <51BF5F00.90705@jitsi.org>
Date: Mon, 17 Jun 2013 21:09:52 +0200
From: Emil Ivov <emcho@jitsi.org>
Organization: Jitsi
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130509 Thunderbird/17.0.6
MIME-Version: 1.0
To: Martin Thomson <martin.thomson@gmail.com>
References: <CAJrXDUHdoxLTsofiwLBdwBNnCCkCBgjSdbmLaXrNEPODMrsSVA@mail.gmail.com> <CABkgnnUmRpanfpwryyiCUsOdMLzrd74n-4LXaj_AK3aLe0yQ8Q@mail.gmail.com>
In-Reply-To: <CABkgnnUmRpanfpwryyiCUsOdMLzrd74n-4LXaj_AK3aLe0yQ8Q@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Gm-Message-State: ALoCoQld9Cl6rZbv6CiN7pMEzby4i7WUZNvYxjvXnppGMVjVhP7G7ElRCEqlvR/RZMfXPvrUxnS4
Cc: "<rtcweb@ietf.org>" <rtcweb@ietf.org>
Subject: Re: [rtcweb] Proposal for a JS API for NoPlan (adding multiple sources without encoding them in SDP)
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 17 Jun 2013 19:10:01 -0000

On 17.06.13, 20:57, Martin Thomson wrote:
> Maybe you'd expect me to be more supportive of something that looked
> so much like CU-RTC-Web.

No, but this doesn't mean you can't at least be constructive ...

> It inherits all the worst properties of JSEP
> (Offer/Answer, SDP editing) with a partial implementation of a clean
> API.
>
> It's comment 22-lite.  It's an abomination.  If you are going to do
> this, do it properly.

So here's your chance: could you please share your view of how this 
would look if done properly?

Emil

>
> On 17 June 2013 05:57, Peter Thatcher <pthatcher@google.com>; wrote:
>> Google is in full support of "Plan B" for encoding multiple media sources in
>> SDP, and would like to see the Plan A vs. Plan B decision resolved soon.
>> Recently, though, a third option, called "NoPlan" has been proposed, but it
>> lacked the details of what a JS API would look like for NoPlan.  Cullen
>> asked to see such an API proposal, and so I have worked with Emil to make
>> one.  He will be adding it to the NoPlan draft, but I will also include it
>> in this email.
>>
>> Again, Google is in full support of "Plan B".  But if Plan A vs. Plan B
>> cannot be decided, then we support NoPlan with the following additions to
>> the WebRTC JS API as an option that allows implementing either Plan A or
>> Plan B  in Javascript.  And even if Plan A vs. Plan B is resolved, these API
>> additions would still be a big improvement for those WebRTC applications
>> that don't use SDP for signalling.
>>
>> It is a bit long because I have added many comments and examples, but the
>> actually additions only include two new methods on PeerConnection and a few
>> new dictionaries.  So don't be overwhelmed :).
>>
>>
>>
>> Intro: This follows the model of createDataChannel, which has a JS method on
>> PeerConnection that makes it possible to add data channels without going
>> through SDP.  Furthermore, just like createDataChannel allows 2 ways to
>> handle neogitation (the "I know what I'm doing;  Here's what I want to send;
>> Let me signal everything" mode and the "please take care of it for me;  send
>> an OPEN message" mode), this also has 2 ways to handle negotiation (the "I
>> know what I'm doing; Here's what I want to send; Let me signal everything"
>> mode and the "please take care of it for me;  send SDP back and forth"
>> mode).
>>
>> Following the success of createDataChannel, this allows simple applications
>> to Just Work and more advanced applications to easily control what they need
>> to.  In particular, it's possible to use this API to implement either Plan A
>> or Plan B.
>>
>> // The following two method are added to RTCPeerConnection
>> partial interface RTCPeerConnection {
>>   // Create a stream that is used to send a source stream.
>>   // The MediaSendStream.description can be used for signalling.
>>   // No media is sent until addStream(MediaSendStream) is called.
>>   LocalMediaStream createLocalStream(MediaStream sourceStream);
>>
>>   // Create a stream that is used to receive media from the remote side,
>>   // given the parameters signalled from MedaiSendStream.description.
>>   MediaStream createRemoteStream(MediaStreamDescription description);
>> }
>>
>>
>> interface LocalMediaStream implements MediaStream {
>>    // This can be changed at any time, but especially before calling
>>    // PeerConnection.addStream
>>    attribute MediaStreamDescription description;
>> }
>>
>>
>> // Represents the parameters used to either send or receive a stream
>> // over a PeerConnection.
>> dictionary MediaStreamDescription {
>>    MediaStreamTrackDescription[] tracks;
>> }
>>
>>
>> // Represents the parameters used to either send or receive a track over //
>> a PeerConnection.  A track has many "flows", which can be grouped
>> // together.
>> dictionary MediaStreamTrackDescription {
>>    // Same as the MediaStreamTrack.id
>>    DOMString id;
>>
>>    // Same as the MediaStreamTrack.kind
>>    DOMString kind;
>>
>>    // A track can have many "flows", such as for Simulcast, FEC, etc.
>>    // And they can be grouped in arbitrary ways.
>>    MediaFlowDescription[] flows;
>>    MediaFlowGroup[] flowGroups;
>> }
>>
>> // Represents the parameters used to either send or receive a "flow"
>> // over a PeerConnection.  A "flow" is a media that arrives with a
>> // single, unique SSRC.  One to many flows together make up the media
>> // for a track.  For example, there may be Simulcast, FEC, and RTX
>> // flows.
>> dictionay MediaFlowDescription {
>>    // The "flow id" must be unique to the track, but need not be unique
>>    // outside of the track (two tracks could both have a flow with the
>>    // same flow ID).
>>    DOMString id;
>>
>>    // Each flow can go over its own transport.  If the JS sets this to a
>>    // transportId that doesn't have a transport setup already, the
>>    // browser will use SDP negotiation to setup a transport to back that
>>    // transportId.  If This is set to an MID in the SDP, then that MID's
>>    // transport is used.
>>    DOMString transportId;
>>
>>    // The SSRC used to send the flow.
>>    unsigned int ssrc;
>>
>>    // When used as receive parameters, this indicates the possible list
>>    // of codecs that might come in for this flow.  For exmample, a given
>>    // receive flow could be setup to receive any of OPUS, ISAC, or PCMU.
>>    // When used as send parameters, this indicates that the first codec
>>    // should be used, but the browser can use send other codecs if it
>>    // needs to because of either bandwidth or CPU constraints.
>>    MediaCodecDescription[] codecs;
>> }
>>
>>
>> dictionary MediaFlowGroup {
>>    DOMString type;  // "SIM" for Simulcast, "FEC" for FEC, etc
>>    DOMString[] flowids;
>> }
>>
>> dictionary MediaCodecDescription {
>>    unsigned byte payloadType;
>>    DOMString name;
>>    unsigned int? clockRate;
>>    unsigned int? bitRate;
>>    // A grab bag of other fmtp that will need to be further defined.
>>    MediaCodecParam[] params;
>> }
>>
>> dictionary MediaCodecParam {
>>    DOMString key;
>>    DOMString value;
>> }
>>
>>
>> Notes:
>>
>> - When LocalMediaStreams are added using addStream, onnegotiatedneeded is
>> not called, and those streams are never reflected in future SDP exchanges.
>> Indeed, it would be impossible to put them in the SDP without first
>> resolving if that would be Plan A SDP or Plan B SDP.
>>
>> - Just like piles of attributes would need to be defined for Plan A and for
>> Plan B, similar attributes would need to be defined here (Luckily,  much
>> work has already been done figuring out what those parameters are :).
>>
>>
>> Pros:
>>
>> - Either Plan A or Plan B or could be implemented in Javascript using this
>> API
>> - It exposes all the same functionality to the Javascript as SDP, but in a
>> much nicer format that is much easier to work with.
>> - Any other signalling mechanism, such as Jingle or CLUE could be
>> implemented using this API.
>> - There is almost no risk of signalling glare.
>> - Debugging errors with misconfigured descriptions should be much easier
>> with this than with large SDP blobs.
>>
>>
>> Cons:
>>
>> - Now there are two slightly different ways to add streams: by creating a
>> LocalMediaStream first, and not.  This is, however, analogous to setting
>> "negotiated: true" in createDataChannel.  On way is "Just Work", and the
>> other is more advanced control.
>>
>> - All the options in MediaCodecDescription are a bit complicated.  Really,
>> this is only necessary because Plan A requires being able to specify codec
>> parameters per SSRC, and set each flow on different transports.  If we did
>> not have this requirement, we could simplify.
>>
>>
>> Example Usage:
>>
>> // Imagine I have MyApp, handles creating a PeerConnection,
>> // signalling, and rendering streams.  This is how the new API could be
>> // used.
>> var peerConnection = MyApp.createPeerConnection();
>>
>> // On sender side:
>> var stream = MyApp.getMediaStream();
>> var localStream = peerConnection.createSendStream(stream);
>> sendStream.description = MyApp.modifyStream(localStream.description)
>> MyApp.signalAddStream(localStream.description, function(response)) {
>>    if (!response.rejected) {
>>      // Media will not be sent.
>>      peerConnection.addStream(localStream);
>>    }
>> }
>>
>> // On receiver side:
>> MyApp.onAddStreamSignalled = function(streamDescription) {
>>    var stream = peerConnection.createReceiveStream(streamDescription);
>>    MyApp.renderStream(stream);
>> }
>>
>>
>> // In this exchange, the MediaStreamDescription signalled from the
>> // sender to the receiver may have looked something like this:
>>
>> {
>>    tracks: [
>>    {
>>      id: "audio1",
>>      kind: "audio",
>>      flows: [
>>      {
>>        id: "main",
>>        transportId: "transport1",
>>        ssrc: 1111,
>>        codecs: [
>>        {
>>          payloadType: 111,
>>          name: "opus",
>>          // ... more codec details
>>        },
>>        {
>>          payloadType: 112,
>>          name: "pcmu",
>>          // ... more codec details
>>        }]
>>     }]
>>   },
>>   {
>>      id: "video1",
>>      kind: "video",
>>      flows: [
>>      {
>>        id: "sim0",
>>        transportId: "transport2",
>>        ssrc: 2222,
>>        codecs: [
>>        {
>>          payloadType: 122,
>>          name: "vp8"
>>          // ... more codec details
>>        }]
>>     },
>>     {
>>       id: "sim1",
>>       transportId: "transport2",
>>       ssrc: 2223,
>>       codecs: [
>>       {
>>         payloadType: 122,
>>         name: "vp8",
>>         // ... more codec details
>>       }]
>>     },
>>     {
>>       id: "sim2",
>>       transportId: "transport2",
>>       ssrc: 2224,
>>       codecs: [
>>       {
>>         payloadType: 122,
>>         name: "vp8",
>>         // ... more codec details
>>       }]
>>     },
>>
>>     {
>>       id: "sim0fec",
>>       transportId: "transport2",
>>       ssrc: 2225,
>>       codecs: [
>>       {
>>         payloadType: 122,
>>         name: "vp8",
>>         // ...
>>       }]
>>     }],
>>     flowGroups: [
>>     {
>>       semantics: "SIM",
>>       ssrcs: [2222, 2223, 2224]
>>     },
>>     {
>>       semantics: "FEC",
>>       ssrcs: [2222, 2225]
>>     }]
>>   }]
>> }
>>
>>
>> Constructive feedback is welcome :).
>>
>> _______________________________________________
>> rtcweb mailing list
>> rtcweb@ietf.org
>> https://www.ietf.org/mailman/listinfo/rtcweb
>>
> _______________________________________________
> rtcweb mailing list
> rtcweb@ietf.org
> https://www.ietf.org/mailman/listinfo/rtcweb
>

-- 
https://jitsi.org