Re: [rtcweb] Comments on H.264 and VP8 performance comparisons

Bo Burman <bo.burman@ericsson.com> Tue, 22 October 2013 14:20 UTC

Return-Path: <bo.burman@ericsson.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2FE4B11E81B3 for <rtcweb@ietfa.amsl.com>; Tue, 22 Oct 2013 07:20:47 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.249
X-Spam-Level:
X-Spam-Status: No, score=-6.249 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, HELO_EQ_SE=0.35, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NE0UCeX3Z7jq for <rtcweb@ietfa.amsl.com>; Tue, 22 Oct 2013 07:20:42 -0700 (PDT)
Received: from mailgw2.ericsson.se (mailgw2.ericsson.se [193.180.251.37]) by ietfa.amsl.com (Postfix) with ESMTP id DCCA211E83D8 for <rtcweb@ietf.org>; Tue, 22 Oct 2013 07:20:41 -0700 (PDT)
X-AuditID: c1b4fb25-b7eff8e000000eda-f4-526689b8c894
Received: from ESESSHC005.ericsson.se (Unknown_Domain [153.88.253.124]) by mailgw2.ericsson.se (Symantec Mail Security) with SMTP id 53.52.03802.8B986625; Tue, 22 Oct 2013 16:20:40 +0200 (CEST)
Received: from ESESSMB105.ericsson.se ([169.254.5.4]) by ESESSHC005.ericsson.se ([153.88.183.33]) with mapi id 14.02.0328.009; Tue, 22 Oct 2013 16:20:40 +0200
From: Bo Burman <bo.burman@ericsson.com>
To: Harald Alvestrand <harald@alvestrand.no>, "rtcweb@ietf.org" <rtcweb@ietf.org>
Thread-Topic: [rtcweb] Comments on H.264 and VP8 performance comparisons
Thread-Index: Ac7JIe76OhlqjGwCQFaenvACTkQ0VwF2Vc2AAA12J5A=
Date: Tue, 22 Oct 2013 14:20:39 +0000
Message-ID: <BBE9739C2C302046BD34B42713A1E2A22DFC7383@ESESSMB105.ericsson.se>
References: <BBE9739C2C302046BD34B42713A1E2A22DF9F8D2@ESESSMB105.ericsson.se> <52664A39.5040105@alvestrand.no>
In-Reply-To: <52664A39.5040105@alvestrand.no>
Accept-Language: sv-SE, en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [153.88.183.20]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrPLMWRmVeSWpSXmKPExsUyM+Jvje6OzrQgg6trzC2O9XWxWaz9187u wORxZcIVVo8lS34yBTBFcdmkpOZklqUW6dslcGUse/WOpWCJWcWVJ4YNjIu0uxg5OSQETCSu 71rLBmGLSVy4tx7I5uIQEjjMKDF/xg1mCGcRo8SrWS8ZQarYBDQk5u+4C2aLCARL9D5/D2YL C7hL9N6aCNTNART3kGhbrgBRYiWx995SJhCbRUBVYsftGawgNq+Ar8Srd8fA4kICBRK7/r5m B7E5BXQlehasBxvJKCArcf/7PRYQm1lAXOLWk/lMEIcKSCzZc54ZwhaVePn4HyuErShxdfpy Joh6HYkFuz+xQdjaEssWvmaG2CsocXLmE5YJjKKzkIydhaRlFpKWWUhaFjCyrGJkz03MzEkv N9rECIyEg1t+q+5gvHNO5BCjNAeLkjjvh7fOQUIC6YklqdmpqQWpRfFFpTmpxYcYmTg4pRoY Rfb/e8F8aMnTCQl7s/dphwVNnS+qlmfYFeH2922MUcuEvbqVgaUaeg3z78hvVZzb2Sm95KCz Vlp0T+6vqM+zGZ4rytUIHhU+/Jldur5i7rOkqdP0n+nuuWXZ+vOjQVKsaNERrpVPzu0NeP7W MuPSY9/5ga+ickou1352XqxTektg0ofsUq5oJZbijERDLeai4kQALANI9lICAAA=
Subject: Re: [rtcweb] Comments on H.264 and VP8 performance comparisons
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 22 Oct 2013 14:20:47 -0000

Hi all,

We understand that there are many people that are interested in a VP8 vs x264 comparison that includes rate controls.

We therefore performed a quick study of the VP8 and x264 rate controls to see whether there were any major differences between them in the way they select quantizers. We found that VP8 uses what can be described as QP-toggling: By lowering the QP of, say, every 6th frame, you increase the quality of these markedly. The following frames also benefit from the increased quality (even though they have to make do with less bits than before) since they can predict from the high-quality frame. 

However, we found that the rate-control of x264 does not QP-toggle in this fine granular way, and x264 may therefore be at a disadvantage when comparing against VP8. So we ran a test using a modified version of x264 to allow QP-toggling during rate control. The modification is very simple and a patch for the x264 code is found in the bottom of this email.

With this change, H.264 baseline (implemented using the modified x264 with rate control turned on) performs equally to VP8 with rate-control on; the difference is 1.3% in favor of VP8, but if you swap the order of the two codecs in draw_graphs.sh, H.264 baseline wins with 1.6%. This result is thus a tie between VP8 and H.264 baseline, and this is consistent with our previous fixed-QP test, and far from the (at least) 13% advantage for VP8 stated by Google.

The modification is simple and it may well be possible to improve x264 by more sophisticated modifications. But pursuing this track is what we oppose, we do not want comparisons where the outcome depends on how well you modify a particular rate control mechanism - the rate control is  codec-agnostic after all. To avoid this we still maintain our view that codec comparisons should be done using fixed QP settings, which is also the practice in MPEG and VCEG.

We used the test parameters from Google's test repository  https://git.chromium.org/git/webm/vpx_codec_comparison.git, the version ada7d7937a54e47fc18e2e0a1287aea29dc5d1c5.

Here are the x264 and comparison script patches:
--

diff --git a/encoder/ratecontrol.c b/encoder/ratecontrol.c
index 1a39a49..2cf256f 100644
--- a/encoder/ratecontrol.c
+++ b/encoder/ratecontrol.c
@@ -1707,7 +1707,18 @@ int x264_ratecontrol_mb_qp( x264_t *h )
{
     x264_emms();
     float qp = h->rc->qpm;
-    if( h->param.rc.i_aq_mode )
+
+       if (h->sh.i_type != SLICE_TYPE_I){
+          if ((h->fenc->i_poc % 12 )== 0)
+              qp -= 3.0;
+          else
+              qp += 0.0;
+   
+       } else {
+          qp -= 5.0;
+       }
+
+    if( h->param.rc.i_aq_mode)
     {
          /* MB-tree currently doesn't adjust quantizers in unreferenced frames. */
         float qp_offset = h->fdec->b_kept_as_ref ? h->fenc->f_qp_offset[h->mb.i_mb_xy] : h->fenc->f_qp_offset_aq[h->mb.i_mb_xy];


draw_graphs.sh:

change 
./visual_metrics.py metrics_template.html "*0.txt" stats/h264 stats/vp8  > vp8_vs_h264_quality.html
to
./visual_metrics.py metrics_template.html "*0.txt" stats/vp8 stats/h264  > vp8_vs_h264_quality2.html

Best Regards,
Bo Burman

> -----Original Message-----
> From: rtcweb-bounces@ietf.org [mailto:rtcweb-bounces@ietf.org] On Behalf Of Harald Alvestrand
> Sent: den 22 oktober 2013 11:50
> To: rtcweb@ietf.org
> Subject: Re: [rtcweb] Comments on H.264 and VP8 performance comparisons
> 
> On 10/14/2013 11:12 PM, Bo Burman wrote:
> > Hi all,
> >
> > We would like to counter Google's suggestion that our test has only "demonstrated that it is possible to reduce VP8's
> performance" (updated draft on VP8 http://datatracker.ietf.org/doc/draft-alvestrand-rtcweb-vp8/).
> >
> > In fact, what we did in our test was mostly undoing some very peculiar
> > x264 settings made by Google in their test from April 3. By instead
> > using the x264 settings Google themselves proposed in their earlier
> > test (from March 12), and removing threading, the difference went down
> > from 41% to 16%. (This is without touching the VP8 parameters.)
> >
> > The last change we made was to remove the rate control from the comparison, something that is standard practice in
> the world of video standardization. This involved changing both the x264 and VP8 parameters. After that, the difference
> went down to -1%.
> >
> > In summary, the following steps were taken in our comparison:
> >
> > 1) Downloading the latest software: 41% became 36%
> > 2) Removing threading: 26%
> > 3) Removing bit padding: 18%
> > 4) Removing other differences between Google's March 12 and April 3rd
> > tests: 16%
> > 5) Removing rate controller: -1%
> >
> >
> Just a quick update on this - I did not manage to get a new draft ready before the deadline, so I'll have to resort to
> sending email:
> 
> I applied steps 1), 2) and 3) to the repository mentioned in the draft.
> The numbers I got were different, but significant. Below are the
> VP8-to-x264 differences I encountered each step of the way.
> 
> - Master branch before October 15: Difference 71.52%
> - Updating x264 from 198a7ea (aug 16 2012) to c832fe (March 1 2013):
> Difference 71.44%
>    (I could not go beyond this because yasm 1.2 was required for newer versions).
> - Adding the --thread 1 parameter: Difference 26.11%
> - Removing the --nai-hrd=cbr parameter that was suggested by x264
> people: Difference 13.87%
> - Removing vbv-maxrate and vbv-init 0.8: Difference 12.74%
> 
> I did not shorten the clips to 10 seconds, nor did I try to control rate by constant cq instead of a rate controller; if Bo's
> numbers are right, this shouldn't matter.
> 
> I did try removing "vbv-bufsize ${rate}", since this was the remaining difference I could find for Bo's point 4 "other
> differences", but that was actually harmful (difference increased to 19.05%), so I put it back.
> 
> I checked this in to the repository - the commit is here:
> 
> http://git.chromium.org/gitweb/?p=webm/vpx_codec_comparison.git;a=commitdiff;h=ada7d7937a54e47fc18e2e0a1287
> aea29dc5d1c5
> 
> 
> This number does not fit the impression the video codec team has from other tests - they think
> VP8 can do a lot more than 13% better than baseline - but at the draft deadline, we had not found the set of VP8
> parameters that showed this for this particular clip set.
> 
> Commentary: I'm surprised at x264's choice of default value for the --thread parameter. Accepting a 26% bitrate hit
> seems like a large price to pay for going faster by default. Is there a bug here?
> 
> _______________________________________________
> rtcweb mailing list
> rtcweb@ietf.org
> https://www.ietf.org/mailman/listinfo/rtcweb