Re: [rtcweb] Multiple videos in one MediaStream (Re: MediaStream Label and CNAME)

"Olle E. Johansson" <oej@edvina.net> Tue, 13 September 2011 14:36 UTC

Return-Path: <oej@edvina.net>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 29CC321F8487 for <rtcweb@ietfa.amsl.com>; Tue, 13 Sep 2011 07:36:31 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.227
X-Spam-Level:
X-Spam-Status: No, score=-2.227 tagged_above=-999 required=5 tests=[AWL=0.022, BAYES_00=-2.599, HELO_EQ_SE=0.35]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id R5RQh4HAdg-n for <rtcweb@ietfa.amsl.com>; Tue, 13 Sep 2011 07:36:30 -0700 (PDT)
Received: from smtp7.webway.se (smtp7.webway.se [212.3.14.205]) by ietfa.amsl.com (Postfix) with ESMTP id E1DE321F86EE for <rtcweb@ietf.org>; Tue, 13 Sep 2011 07:36:24 -0700 (PDT)
Received: from [192.168.40.44] (ns.webway.se [87.96.134.125]) by smtp7.webway.se (Postfix) with ESMTPA id BD0ED754BCE5; Tue, 13 Sep 2011 14:38:29 +0000 (UTC)
Mime-Version: 1.0 (Apple Message framework v1244.3)
Content-Type: text/plain; charset=windows-1252
From: "Olle E. Johansson" <oej@edvina.net>
In-Reply-To: <4E6F6963.9090702@alvestrand.no>
Date: Tue, 13 Sep 2011 16:38:29 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <D2889951-B3E0-48BE-9CF0-327776298122@edvina.net>
References: <4E6F17AB.4000005@ericsson.com> <4E6F6963.9090702@alvestrand.no>
To: Harald Alvestrand <harald@alvestrand.no>
X-Mailer: Apple Mail (2.1244.3)
Cc: rtcweb@ietf.org
Subject: Re: [rtcweb] Multiple videos in one MediaStream (Re: MediaStream Label and CNAME)
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 13 Sep 2011 14:36:31 -0000

13 sep 2011 kl. 16:32 skrev Harald Alvestrand:

> On 09/13/11 10:43, Magnus Westerlund wrote:
>> WG,
>> (As an individual contributor)
>> 
>> 
>> There has been some discussion as result of the presentation of
>> terminology in the RTCWEB Interim meeting last Thursday. The biggest
>> question was why CNAME can't map to MediaStream label. Below we
>> clarify why we think CNAME and label are separate entities.
>> 
>> One part in this reasoning has to do with the current definition of
>> ’media resource’
>> (<http://dev.w3.org/html5/spec/Overview.html#media-resource>) and media
>> elements of html5. The ‘media resource’ could be a file, or, more
>> relevant to this discussion, a MediaStream. In that usage only a single
>> video track can be played simultaneously and in sync with one or more
>> audio tracks.
>> 
>> Thus unless we modify an existing semantics the only way of playing
>> multiple video tracks in sync with one or more audio tracks is to have
>> multiple MediaStream objects.
> Sorry, I think this can be handled with the existing API without any issue.
> Apologies to those who know JavaScript better than me, but this is a handler
> that can handle a new incoming video stream
> 
> NewStreamHandler(stream) {
>   myMultiVideoStream = stream
>   myFirstVideo = new MediaStream(myMultiVideoStream)
>   myFirstVideo.videoTracks.select(1)
>   oneVideoObject.setSource(myFirstVideo.getUrl())
>   mySecondVideo = new MediaStream(myMultiVideoStream)
>   mySecondVideo.videoTracks.select(2)
>   anotherVideoObject.setSource(mySecondVideo.getUrl())
> }
> 
> I think the API can be improved, but the improvements are unlikely to lose this functionality.
> (Remember that the API objects are the control surfaces on the stream - the video data will likely go straight from the incoming buffer, through the codec for decoding, and blitted onto the screen; the "copying" from one MediaStream to another MediaStream is purely conceptual.)
> 
> I don't think it makes sense to discuss the other points before getting this one put to rest.
> 
>     
Just to add to the stew:

 - For each stream  (video, text, audio) you have one outbound and at least one inbound
 - If we are going to support SDP and SIP, you will end up with multiple inbound streams - one at a time or at the same time

Forking may lead to multiple incoming streams for one outbound "call". Early media may lead to one incoming stream, which may be replaced by another endpoint at "answer time".

I think one important question here is if we are going to allow this at all. If we follow the SIP model we just have to. 

If possible, I would like to see a restriction in rtcweb so that we push this complexity to the server, far away from the browser.

/O