Re: [rtcweb] Support of video with different resolutions

[Randell Jesup <[e-mail]>]

On 12/21/2012 6:27 PM, Gunnar Hellström wrote:
> On 2012-12-21 16:40, Bernard Aboba wrote:
>> SVC supports multiple scalability mechanisms: temporal, spatial and 
>> quality. Temporal is the most widely implemented in software (at 
>> least for H.264/SVC), although there is an open source implementation 
>> that supports all modes. In hardware, support for spatial or quality 
>> scaling is not common. As a result, a browser on a mobile device 
>> could have difficulty complying with a mandate to support spatial 
>> scaling within the encoder.
>>
> This is sad. For good usability of video, maintained frame rate is 
> usually much more important than maintained spatial resolution. E.g. 
> for sign language or lip reading usage with a single person in image, 
> a frame rate under 20 fps introduces loss of language contents, and 
> requires the users to try to fill in the gaps by imagination, while 
> spatial resolution reduction down to QCIF causes much less harm to 
> language perception possibilities.

Strong agreement. High (and consistent!) frame rate is critical to VRS 
and sign-language - and for hearing people, it 'feels' much more like 
talking to someone if the video framerate is kept >20 (and preferably 
 >25), with perfect lip-sync. This is based on the experience I and 
Maire Reavy had at Worldgate, where we provided videophones for VRS 
usage. 30FPS QCIF works pretty well for sign-language, and better 
generally than 20FPS CIF, even though CIF is 4x the number of pixels.

Note that there's no reason a browser in a mobile device couldn't 
maintain 30fps, and if congestion indicates it needs to reduce bandwidth 
too much for 'good quality' 30FPS, it can instead reduce the resolution. 
The only time I'd value resolution over framerate is when viewing 
graphics or the equivalent. SVC certainly isn't needed here for 1-1 
communication; a main advantage of SVC is to allow a node in the network 
(i.e. conf. server/MCU) to subset down a stream for one recipient while 
providing higher-resolution streams to others while avoiding 
decode/re-encode on the server. Of course, this was the use-case that 
started this.

In a corollary, frame rate should not drop when there's a lot of 
movement, or should only drop momentarily. (And I mean very momentarily, 
like dropping a single frame when there's a big jump in motion from the 
last frame, then resuming 30FPS at higher quantization or higher bitrate.)

However, Christer's original question said:
"However, the “local transcoding” alternative will consume lots of 
resources, and as far as I know none of the proposed MTI video codecs 
support SVC. "

This is I believe is at least partially incorrect. From 
blog.webmproject.com, 1/27/2012, "VP8 Codec SDK "Duclair" Released":

    This release introduces substantial new VP8 encoder features that
    are especially useful for real-time use cases such as live streaming
    and videoconferencing.

      * Temporal scalability produces a video stream that can be
        decimated to different frame rates, with independent rate
        targeting for each substream.
      * Multiframe postprocessing can make visual quality more
        consistent in the presence of frames that are of substantially
        different quality than the surrounding frames, as in the
        temporal scalability case and in some forced keyframe scenarios.
      * Multiple-resolution encoding enables simultaneous encoding of
        the same content at different resolutions, resulting in much
        faster encoding than processing them separately.

This is both multicast and temporal scalability.

-- 
Randell Jesup
[e-mail]