Re: [rtcweb] VP8 vs H.264 - subjective evaluation

[Harald Alvestrand <[e-mail]>]

On 10/23/2013 01:46 AM, Krasimir Kolarov wrote:
> To be very clear for the IETF community:
>
> As Harald’s slide deck notes on Slide 2, these subjective test results were not presented nor discussed at MPEG meetings. Hence they only reflect the opinion of the VP8 proponents.

Well.... strictly speaking, the results reflect the work of Vittorio 
Baronici.
The methods reflect the methods chosen by MPEG.
The presentation was made by a VP8 proponent.

>
> The only subjective tests that were presented at the meeting compared VP8 and IVC (the Internet Video Coding proposal considered by Mpeg, which is different than AVC Constrained Baseline).
> Those tests raised a lot of controversy during discussions at the meeting and in fact some of the results had to be excluded because of significant differences in comparison conditions (rate control, etc.).

Krasimir, I think you're mixing up two debates here (easy, since there 
have been two sets of subjective evaluation tests).

The use of rate control was accepted by MPEG, despite protests by some 
of the members, since this is part of the design of VP8 in a way that it 
was never part of the design of H.264.

No result was ever excluded in the IVC to VP8 comparision. After 
discussion, we all agreed that the results needed to be presented with a 
horizontal axis representing actual rate, not target rate; this has been 
done in the slides I sent out.

Two results were dropped from the presentation of the VP8 results, which 
were not presented as a comparision with AVC High profile, due to issues 
with achieving very low bitrates.

>   This is not unlike recent discussions on this reflector about comparisons of VP8 and H.264 Constrained Baseline.

Agreed. In both cases, the protestations that rate control is a problem 
reflect a position that MPEG did not come to consensus on. MPEG accepted 
the results that were produced using rate control.

>
> Krasimir
>
> On Oct 22, 2013, at 3:24 AM, Harald Alvestrand <[e-mail]> wrote:
>
>> In my VP8 advocate draft, I referred to a subjective evaluation test
>> (test with
>> actual human viewers) done between VP8 and H.264 Baseline by a neutral
>> (non-Google) laboratory.
>>
>> The attached presentation is the writeup of the results of that test.
>>
>> -- 
>> Surveillance is pervasive. Go Dark.
>>
>> <VP8 vs AVC Baselinev2.pdf>_______________________________________________
>> rtcweb mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/rtcweb
>