Re: [rtcweb] VP8 vs H.264 - subjective evaluation

[Harald Alvestrand <[e-mail]>]

On 10/24/2013 10:49 AM, Bo Burman wrote:
> It is really difficult to at all comment on this since there is far too little data available. There is no way to review the settings and parameters used, and of course no way to repeat the tests, and hopefully our previous inputs related to objective quality evaluation has made it clear to everyone that parameter settings make a huge difference.

Bo, your organization is an MPEG member, so you can get the streams, and
the scripts used to produce them, any day you want.
I've sent you the login details to the server in separate mail.

The VP8 parameters were the ones used for the VP8 vs IVC comparision
presented in MPEG input documents to the Geneva MPEG meeting; I think
you will find the actual command lines either in the input documents or
in the attachments.

The AVC streams used were the ones produced for the initial MPEG
"royalty-free video codec" call for proposals 2 years back, using the JM
encoder; I don't know who produced them, it might even have been
Ericsson. I'd expect that you'd find the encoding parameters used in
those documents.

If you want to repeat the tests, feel very free to hire your own
independent laboratory to do so - Vittorio's evaluation methods are very
well known in the MPEG community.

We have absolutely no interest in keeping anything secret here - the
only reason it's not all in the open is because of MPEG rules about
access to documents and test sequences!

>
> It can also be noted that even if the results are assumed to be correct (which there is currently no way to verify) the conclusions drawn seem incorrect. E.g., it is stated that "In 7/10 sequences tested VP8 LD was clearly better than AVC constrained baseline", but when examining the plots the confidence intervals seem to be overlapping.

I'm not sure what you are looking at - could you identify the slides
where you see overlapping confidence intervals between the red line (VP8
LD) and the green line (AVC constrained baseline)?

>
> Without any more information it can only be concluded that this test has very little value.

As I've noted in another thread, discussions about quality seem to have
very little value anyway, since there's nobody who's said that they will
change their mind if they get some new quality information. But we had
this evaluation already, so sharing information with the community
seemed to be the community-minded thing to do.

-- 
Surveillance is pervasive. Go Dark.