Re: [rtcweb] More H.264 vs VP8 tests

[Harald Alvestrand <[e-mail]>]

First of all, Bo, thanks for running the tests, and for making the 
results fully open!

On 06/24/2013 01:33 PM, Stefan Håkansson LK wrote:
> On 2013-06-22 18:54, Leon Geyser wrote:
>> Hi Bo,
>>
>> Thanks for the additional testing done.
>>
>> I am not 100% why we have to test with fixed qp-levels when WebRTC is
>> going to be used over the internet. I would assume that WebRTC would be
>> used over the internet...
>> My understanding of rate-control might be completely wrong. Why test
>> modes that might not even be used in real-world scenarios?
> Hi Leon (responding for Bo who is enjoying his vacation),
>
> you are absolutely right, these codecs are going to be used with
> rate-controllers when deployed in the real world. However, we still do
> not think it is a good idea having rate control turned on when comparing
> them. The reason is that the rate controller influences the quality so
> much that it is very hard to get any comparable signal if you include
> them. Let me give you one example; let's say that you use the same
> codec, but two different rate controllers. Rate control A is very
> liberal with its bits in the beginning of the sequence, whereas rate
> control B is very frugal. On a video conferencing type sequence, where
> the camera is static and not much is happening, rate control A will win
> by a huge margin over rate control B. The reason is that the first frame
> will be encoded very well and this data can be used during the remainder
> of the sequence. Rate control B will not be able to stand a chance, even
> though it has more bits for the remaining images.  And this is with the
> same video codec!

This would seem to me to be an argument for including the rate 
control.... what we are looking for is the codec where people can 
demonstrate the best performance for our specific situations, not the 
codec that is best able to fulfil a set of conditions that are 
inappropriate for Real Life.

For the example you give, the bit division between the I-frame/key-frame 
and the subsequent frames is not just influenced by the chosen Q value - 
it's also influenced by codec design.

What will happen in a video conference scenario if someone uses this 
rate packing too much is that the I-frame takes a long time to transmit, 
because the available bit rate is in the ramp-up phase; if we wanted to 
make realistic scenario settings, we should probably draw constraints 
around the delay in delivering the second frame under specified 
bandwidth conditions.

But on the other hand, people may find a delay to the second frame quite 
acceptable, and your "rate control A" is indeed the one that is most 
reasonable to use.

>
> We could of course try to make sure that the two rate controllers have
> similar settings; however, since they are not identical they cannot be
> made to work exactly the same. Even small changes in rate control
> settings could have a huge effect on the end result.
>
> Rather than trying to make two pieces of software work the same, it is
> much easier to just shut off the rate controllers. This gives a direct
> measurement of the relative strengths of the codecs. This is also the
> reason why organizations such as JC-TVC have used this method to compare
> their codecs.

And it's going to be discussed there.... I'll be late for the Berlin 
IETF meeting because I have to be at the MPEG meeting in Vienna to 
present Google's formal submission of VP8 to MPEG. The use or non-use of 
variable QP will probably be an item of debate .... again.

The IETF has not started down the path of defining test conditions under 
which it wishes to measure codec performance; I think we do have rough 
consensus that the technical differences are not so great as to make a 
decisive difference between the proposed codecs - while the IPR 
differences are all too well known to us, and do matter enough to make a 
decisive difference to a lot of us.

                Harald