Re: [rtcweb] Mute implementations (Re: More Comments ondraft-ietf-rtcweb-use-cases-and-requirements-07)

[Stefan Hakansson LK <[e-mail]>]

For the fun of it I thought a little while about what options that are 
currently available with the APIs that are in the W3C docs as is. The 
list is in no way exhaustive, there are other ways to do it.

Perhaps this is not sufficient for what people want to do, but at least 
there are some things you can do already:

1. Replace video with the display of an image
=============================================
One solution: Use the “poster” attribute of the video element. Quote 
from the html spec: “The poster attribute gives the address of an image 
file that the user agent can show while no video data is available.”. We 
can spec it such that if no video track of a MediaStreamis active, that 
counts “as no data available”. This way, as soon as the user “mutes” the 
video, an image will replace it at display. The site can supply the 
image (either a default by the site or one “profile” pic selected by the 
user), or it can be provided over the data channel (as a blob).

Another solution: A signal is sent on the data channel (possibly 
accompanied by the image) on the data channel, the peer’s app overlays 
the image on top of the video element as result (and switches back on 
another signal).

A third (similar) solution: The app at the receiving end listens to 
“onunmute” of the video track in question; once it fires an image is 
overlayed.

A fourth: The peer’s app analyzes the video using canvas, and overlays 
an image when there is no data in the video.

2. Switch between front and rear camera
========================================
One solution: create a MediaStream (e.g. by calling getUserMedia twice 
immediately after each other, or by combining existing tracks into a new 
MediaStream) containing two video tracks (front+back cam). The video 
shown is toggled by enabling/disabling video tracks (the video element 
plays the first active one in the case of several video tracks).

Another: use add/removeTrack to switch video track being sent.

3. Play music on hold
=====================
One solution (audio only case): The site provides an URL to a (music) 
file to play when on hold. The peer’s app registers an event listener 
that listens to “onmute” on the incoming audio track. When “onmute” 
fires, the app changes the source of the audio element to the URL of the 
music file.

One solution if the communication is audio+video, and the intent is to 
just replace the audio (but continue playing video): assume a 
MediaStream with one audio and one video track is used. Play the audio 
in an audio element, the video in a (muted) video element. At on hold, 
replace the source of the audio element with the music file.

A solution if the communication is audio+video, and the intent is to 
replace audio and video: either “mute” the tracks (at sending side); 
receiving end gets “onmute” event and replaces the .src of the video 
element with the URL to a video file, or signal on the data channel 
something that makes the other end replace the src of the video element 
with an URL to a file

Another solution is that the sending end does “removeStream” from 
PeerConnection. This fires “onremovestream” at the receiving side; that 
event can trigger the change of src to the video element. This solution 
frees up resources.

Note that once the recording function has been specced (in such a way 
that the recorded file can be the source of a MediaStream) more options 
will become available.

--Stefan

On 06/17/2012 02:38 AM, Randell Jesup wrote:
> On 6/16/2012 5:58 PM, Jim Barnett wrote:
>> It's capturing the requirement accurately that I'm concerned with.  I
>> think "To be able to change the source of a video stream after its
>> creation" is too broad, in the sense that someone might interpret it to
>> mean that the system could shift from the front to the rear camera
>> without asking permission.  For mute and hold, we want something like
>> "switch to a dummy source and back".  The source that we switch to in
>> cases like this is not a 'first-class' video source, but rather a
>> place-holder.  That's very different from switching to another live
>> source.
>
> Which, by the way, we should be able to do.
>
> MediaStreams need to be able to be repointed, switched, processed, etc,
> OR you need to be able to replace a MediaStream in the inputs to
> PeerConnection with another one transparently without renegotiation (and
> that misses a bunch of uses).
>
> The MediaStream Processing provides a mechanism to do this for both
> audio and video, plus the ability to do more complex operations
> (cross-fades audio-and-video, frame-accurate source switches, etc).
>
> If you don't have or use that, you need the ability to replace a track
> in a MediaStream with another track.
>
> All we're talking about are mechanisms to shuffle existing or creatable
> tracks/streams; a new camera source would require a call to
> getUserMedia() and a dialog.  Once you had front and back, you could
> switch sources without effectively shutting down all media and restarting.
>
> Simple one camera ->  one video stream mechanisms are insufficient for
> application using these features.  If you force people, they'll roll
> their own using canvases, etc.
>
> I want something generic and usable in lots of interesting ways, not a
> few special-purpose pre-coded mechanisms (for example, a
> track.setMuteImage() method).
>