Re: [rtcweb] VP8 vs H.264 - subjective evaluation

Harald Alvestrand <> Wed, 23 October 2013 06:04 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 2645211E82E3 for <>; Tue, 22 Oct 2013 23:04:23 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -110.517
X-Spam-Status: No, score=-110.517 tagged_above=-999 required=5 tests=[AWL=0.082, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id BiLDaATnpRDQ for <>; Tue, 22 Oct 2013 23:04:18 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id F11AB11E8162 for <>; Tue, 22 Oct 2013 23:04:17 -0700 (PDT)
Received: from localhost (localhost []) by (Postfix) with ESMTP id DD36C39E127; Wed, 23 Oct 2013 08:04:15 +0200 (CEST)
X-Virus-Scanned: Debian amavisd-new at
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id sQ6B2JUTbs5H; Wed, 23 Oct 2013 08:04:15 +0200 (CEST)
Received: from [] (unknown []) by (Postfix) with ESMTPSA id 184B839E116; Wed, 23 Oct 2013 08:04:15 +0200 (CEST)
Message-ID: <>
Date: Wed, 23 Oct 2013 08:04:21 +0200
From: Harald Alvestrand <>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.0
MIME-Version: 1.0
To: Krasimir Kolarov <>, "" <>
References: <> <>
In-Reply-To: <>
Content-Type: text/plain; charset="windows-1252"; format="flowed"
Content-Transfer-Encoding: 8bit
Subject: Re: [rtcweb] VP8 vs H.264 - subjective evaluation
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 23 Oct 2013 06:04:23 -0000

On 10/23/2013 01:46 AM, Krasimir Kolarov wrote:
> To be very clear for the IETF community:
> As Harald’s slide deck notes on Slide 2, these subjective test results were not presented nor discussed at MPEG meetings. Hence they only reflect the opinion of the VP8 proponents.

Well.... strictly speaking, the results reflect the work of Vittorio 
The methods reflect the methods chosen by MPEG.
The presentation was made by a VP8 proponent.

> The only subjective tests that were presented at the meeting compared VP8 and IVC (the Internet Video Coding proposal considered by Mpeg, which is different than AVC Constrained Baseline).
> Those tests raised a lot of controversy during discussions at the meeting and in fact some of the results had to be excluded because of significant differences in comparison conditions (rate control, etc.).

Krasimir, I think you're mixing up two debates here (easy, since there 
have been two sets of subjective evaluation tests).

The use of rate control was accepted by MPEG, despite protests by some 
of the members, since this is part of the design of VP8 in a way that it 
was never part of the design of H.264.

No result was ever excluded in the IVC to VP8 comparision. After 
discussion, we all agreed that the results needed to be presented with a 
horizontal axis representing actual rate, not target rate; this has been 
done in the slides I sent out.

Two results were dropped from the presentation of the VP8 results, which 
were not presented as a comparision with AVC High profile, due to issues 
with achieving very low bitrates.

>   This is not unlike recent discussions on this reflector about comparisons of VP8 and H.264 Constrained Baseline.

Agreed. In both cases, the protestations that rate control is a problem 
reflect a position that MPEG did not come to consensus on. MPEG accepted 
the results that were produced using rate control.

> Krasimir
> On Oct 22, 2013, at 3:24 AM, Harald Alvestrand <> wrote:
>> In my VP8 advocate draft, I referred to a subjective evaluation test
>> (test with
>> actual human viewers) done between VP8 and H.264 Baseline by a neutral
>> (non-Google) laboratory.
>> The attached presentation is the writeup of the results of that test.
>> -- 
>> Surveillance is pervasive. Go Dark.
>> <VP8 vs AVC Baselinev2.pdf>_______________________________________________
>> rtcweb mailing list