Re: [rtcweb] Comments on H.264 and VP8 performance comparisons

Harald Alvestrand <harald@alvestrand.no> Tue, 22 October 2013 09:49 UTC

Return-Path: <harald@alvestrand.no>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A78C811E82F0 for <rtcweb@ietfa.amsl.com>; Tue, 22 Oct 2013 02:49:46 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -110.599
X-Spam-Level:
X-Spam-Status: No, score=-110.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z5-dmshV+VqZ for <rtcweb@ietfa.amsl.com>; Tue, 22 Oct 2013 02:49:42 -0700 (PDT)
Received: from eikenes.alvestrand.no (eikenes.alvestrand.no [158.38.152.233]) by ietfa.amsl.com (Postfix) with ESMTP id F242A21F9DDE for <rtcweb@ietf.org>; Tue, 22 Oct 2013 02:49:41 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by eikenes.alvestrand.no (Postfix) with ESMTP id 5508D39E12D for <rtcweb@ietf.org>; Tue, 22 Oct 2013 11:49:41 +0200 (CEST)
X-Virus-Scanned: Debian amavisd-new at eikenes.alvestrand.no
Received: from eikenes.alvestrand.no ([127.0.0.1]) by localhost (eikenes.alvestrand.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id f9qLAfprMInN for <rtcweb@ietf.org>; Tue, 22 Oct 2013 11:49:40 +0200 (CEST)
Received: from [192.168.1.17] (unknown [188.113.88.47]) by eikenes.alvestrand.no (Postfix) with ESMTPSA id 57A7139E062 for <rtcweb@ietf.org>; Tue, 22 Oct 2013 11:49:40 +0200 (CEST)
Message-ID: <52664A39.5040105@alvestrand.no>
Date: Tue, 22 Oct 2013 11:49:45 +0200
From: Harald Alvestrand <harald@alvestrand.no>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.0
MIME-Version: 1.0
To: rtcweb@ietf.org
References: <BBE9739C2C302046BD34B42713A1E2A22DF9F8D2@ESESSMB105.ericsson.se>
In-Reply-To: <BBE9739C2C302046BD34B42713A1E2A22DF9F8D2@ESESSMB105.ericsson.se>
Content-Type: text/plain; charset="ISO-8859-1"; format="flowed"
Content-Transfer-Encoding: 7bit
Subject: Re: [rtcweb] Comments on H.264 and VP8 performance comparisons
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 22 Oct 2013 09:49:46 -0000

On 10/14/2013 11:12 PM, Bo Burman wrote:
> Hi all,
>
> We would like to counter Google's suggestion that our test has only "demonstrated that it is possible to reduce VP8's performance" (updated draft on VP8 http://datatracker.ietf.org/doc/draft-alvestrand-rtcweb-vp8/).
>
> In fact, what we did in our test was mostly undoing some very peculiar x264 settings made by Google in their test from April 3. By instead using the x264 settings Google themselves proposed in their earlier test (from March 12), and removing threading, the difference went down from 41% to 16%. (This is without touching the VP8 parameters.)
>
> The last change we made was to remove the rate control from the comparison, something that is standard practice in the world of video standardization. This involved changing both the x264 and VP8 parameters. After that, the difference went down to -1%.
>
> In summary, the following steps were taken in our comparison:
>
> 1) Downloading the latest software: 41% became 36%
> 2) Removing threading: 26%
> 3) Removing bit padding: 18%
> 4) Removing other differences between Google's March 12 and April 3rd tests: 16%
> 5) Removing rate controller: -1%
>
>
Just a quick update on this - I did not manage to get a new draft ready 
before the deadline, so I'll have to resort to sending email:

I applied steps 1), 2) and 3) to the repository mentioned in the draft. 
The numbers I got were different, but significant. Below are the 
VP8-to-x264 differences I encountered each step of the way.

- Master branch before October 15: Difference 71.52%
- Updating x264 from 198a7ea (aug 16 2012) to c832fe (March 1 2013): 
Difference 71.44%
   (I could not go beyond this because yasm 1.2 was required for newer 
versions).
- Adding the --thread 1 parameter: Difference 26.11%
- Removing the --nai-hrd=cbr parameter that was suggested by x264 
people: Difference 13.87%
- Removing vbv-maxrate and vbv-init 0.8: Difference 12.74%

I did not shorten the clips to 10 seconds, nor did I try to control rate 
by constant cq instead of a rate controller; if Bo's numbers are right, 
this shouldn't matter.

I did try removing "vbv-bufsize ${rate}", since this was the remaining 
difference I could find for Bo's point 4 "other differences", but that 
was actually harmful (difference increased to 19.05%), so I put it back.

I checked this in to the repository - the commit is here:

http://git.chromium.org/gitweb/?p=webm/vpx_codec_comparison.git;a=commitdiff;h=ada7d7937a54e47fc18e2e0a1287aea29dc5d1c5


This number does not fit the impression the video codec team has from 
other tests - they think
VP8 can do a lot more than 13% better than baseline - but at the draft 
deadline, we had not found the
set of VP8 parameters that showed this for this particular clip set.

Commentary: I'm surprised at x264's choice of default value for the 
--thread parameter. Accepting a 26% bitrate hit seems like a large price 
to pay for going faster by default. Is there a bug here?