Re: [rtcweb] Multiple videos in one MediaStream (Re: MediaStream Label and CNAME)

[Harald Alvestrand <[e-mail]>]

On 09/14/11 11:53, Magnus Westerlund wrote:
> On 2011-09-13 16:32, Harald Alvestrand wrote:
>> On 09/13/11 10:43, Magnus Westerlund wrote:
>>> WG,
>>> (As an individual contributor)
>>>
>>>
>>> There has been some discussion as result of the presentation of
>>> terminology in the RTCWEB Interim meeting last Thursday. The biggest
>>> question was why CNAME can't map to MediaStream label. Below we
>>> clarify why we think CNAME and label are separate entities.
>>>
>>> One part in this reasoning has to do with the current definition of
>>> ’media resource’
>>> (<http://dev.w3.org/html5/spec/Overview.html#media-resource>) and media
>>> elements of html5. The ‘media resource’ could be a file, or, more
>>> relevant to this discussion, a MediaStream. In that usage only a single
>>> video track can be played simultaneously and in sync with one or more
>>> audio tracks.
>>>
>>> Thus unless we modify an existing semantics the only way of playing
>>> multiple video tracks in sync with one or more audio tracks is to have
>>> multiple MediaStream objects.
>> Sorry, I think this can be handled with the existing API without any issue.
>> Apologies to those who know JavaScript better than me, but this is a handler
>> that can handle a new incoming video stream
>>
>> NewStreamHandler(stream) {
>>      myMultiVideoStream = stream
>>      myFirstVideo = new MediaStream(myMultiVideoStream)
>>      myFirstVideo.videoTracks.select(1)
>>      oneVideoObject.setSource(myFirstVideo.getUrl())
>>      mySecondVideo = new MediaStream(myMultiVideoStream)
>>      mySecondVideo.videoTracks.select(2)
>>      anotherVideoObject.setSource(mySecondVideo.getUrl())
>> }
>>
>> I think the API can be improved, but the improvements are unlikely to
>> lose this functionality.
>> (Remember that the API objects are the control surfaces on the stream -
>> the video data will likely go straight from the incoming buffer, through
>> the codec for decoding, and blitted onto the screen; the "copying" from
>> one MediaStream to another MediaStream is purely conceptual.)
>>
>> I don't think it makes sense to discuss the other points before getting
>> this one put to rest.
> Harald,
>
> If I read this correctly, what you are doing is to creating two media
> streams based on a first media stream. Thus you have multiple
> mediaStream objects that contains tracks that has a common
> synchronziation context. So from my point of view you have very well
> demonstrated why you need CNAME to be a property associated with the
> tracks in the media stream and not be the label for the mediaStream.
That's a good point.

My thinking when typing this was that stuff that happens inside a 
browser is synchronized "enough" by default, so that we don't need to 
carry the label along in order to keep synchronity between myFirstVideo 
and mySecondVideo. This may or may not be optimistic; input sought.

The audio tracks inside myFirstVideo are declared as synchronized to the 
myFirstVideo video track by virtue of being inside the same MediaStream 
object; this carries over to them being synchronized if myFirstVideo is 
connected to another PeerConnection, if this is allowed (see "recording" 
discussion). (In this case, I think the CNAME should NOT be the same as 
that coming in over the incoming PeerConnection).
> Or how do you see the label for the media stream being handled when you
> create two mediaStream objects from the incomming "stream" that has one
> label?
My original thinking was that a CNAME is created for any MediaStream at 
creation time, so that we're operating with three CNAMEs in the example 
above.

Another possibility is that CNAME is a property of the attachment of a 
MediaStream to a PeerConnection object; this has the advantage that the 
CNAME property is only defined in the cases where we need it, but we 
don't have to carry the baggage of actually generating CNAMEs for 
MediaStreams that are not connected to PeerConnections.