Re: [rtcweb] VP8 vs H.264 - subjective evaluation

Harald Alvestrand <> Thu, 24 October 2013 11:33 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 5D20211E8312 for <>; Thu, 24 Oct 2013 04:33:10 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -110.528
X-Spam-Status: No, score=-110.528 tagged_above=-999 required=5 tests=[AWL=0.071, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id XN1slBfn+Rfa for <>; Thu, 24 Oct 2013 04:33:04 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id 9008B11E8240 for <>; Thu, 24 Oct 2013 04:33:03 -0700 (PDT)
Received: from localhost (localhost []) by (Postfix) with ESMTP id 4588839E18D for <>; Thu, 24 Oct 2013 13:33:00 +0200 (CEST)
X-Virus-Scanned: Debian amavisd-new at
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 2UZxaJ6-XX+o for <>; Thu, 24 Oct 2013 13:32:59 +0200 (CEST)
Received: from [] (unknown []) by (Postfix) with ESMTPSA id 027F539E031 for <>; Thu, 24 Oct 2013 13:32:59 +0200 (CEST)
Message-ID: <>
Date: Thu, 24 Oct 2013 13:32:58 +0200
From: Harald Alvestrand <>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.0
MIME-Version: 1.0
References: <> <> <> <> <>
In-Reply-To: <>
X-Enigmail-Version: 1.5.2
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit
Subject: Re: [rtcweb] VP8 vs H.264 - subjective evaluation
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 24 Oct 2013 11:33:10 -0000

On 10/24/2013 10:49 AM, Bo Burman wrote:
> It is really difficult to at all comment on this since there is far too little data available. There is no way to review the settings and parameters used, and of course no way to repeat the tests, and hopefully our previous inputs related to objective quality evaluation has made it clear to everyone that parameter settings make a huge difference.

Bo, your organization is an MPEG member, so you can get the streams, and
the scripts used to produce them, any day you want.
I've sent you the login details to the server in separate mail.

The VP8 parameters were the ones used for the VP8 vs IVC comparision
presented in MPEG input documents to the Geneva MPEG meeting; I think
you will find the actual command lines either in the input documents or
in the attachments.

The AVC streams used were the ones produced for the initial MPEG
"royalty-free video codec" call for proposals 2 years back, using the JM
encoder; I don't know who produced them, it might even have been
Ericsson. I'd expect that you'd find the encoding parameters used in
those documents.

If you want to repeat the tests, feel very free to hire your own
independent laboratory to do so - Vittorio's evaluation methods are very
well known in the MPEG community.

We have absolutely no interest in keeping anything secret here - the
only reason it's not all in the open is because of MPEG rules about
access to documents and test sequences!

> It can also be noted that even if the results are assumed to be correct (which there is currently no way to verify) the conclusions drawn seem incorrect. E.g., it is stated that "In 7/10 sequences tested VP8 LD was clearly better than AVC constrained baseline", but when examining the plots the confidence intervals seem to be overlapping.

I'm not sure what you are looking at - could you identify the slides
where you see overlapping confidence intervals between the red line (VP8
LD) and the green line (AVC constrained baseline)?

> Without any more information it can only be concluded that this test has very little value.

As I've noted in another thread, discussions about quality seem to have
very little value anyway, since there's nobody who's said that they will
change their mind if they get some new quality information. But we had
this evaluation already, so sharing information with the community
seemed to be the community-minded thing to do.

Surveillance is pervasive. Go Dark.