Re: [rtcweb] Comments on H.264 and VP8 performance comparisons

Bo Burman <bo.burman@ericsson.com> Tue, 22 October 2013 16:07 UTC

Return-Path: <bo.burman@ericsson.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B169811E81D4 for <rtcweb@ietfa.amsl.com>; Tue, 22 Oct 2013 09:07:24 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.124
X-Spam-Level:
X-Spam-Status: No, score=-4.124 tagged_above=-999 required=5 tests=[AWL=-2.125, BAYES_00=-2.599, J_CHICKENPOX_31=0.6]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WvvACcniL4CS for <rtcweb@ietfa.amsl.com>; Tue, 22 Oct 2013 09:07:20 -0700 (PDT)
Received: from sesbmg20.ericsson.net (sesbmg20.ericsson.net [193.180.251.56]) by ietfa.amsl.com (Postfix) with ESMTP id 5B45811E81C4 for <rtcweb@ietf.org>; Tue, 22 Oct 2013 09:07:09 -0700 (PDT)
X-AuditID: c1b4fb38-b7fcf8e0000062b8-75-5266a2ab8513
Received: from ESESSHC015.ericsson.se (Unknown_Domain [153.88.253.124]) by sesbmg20.ericsson.net (Symantec Mail Security) with SMTP id E9.F6.25272.BA2A6625; Tue, 22 Oct 2013 18:07:08 +0200 (CEST)
Received: from ESESSMB105.ericsson.se ([169.254.5.4]) by ESESSHC015.ericsson.se ([153.88.183.63]) with mapi id 14.02.0328.009; Tue, 22 Oct 2013 18:07:07 +0200
From: Bo Burman <bo.burman@ericsson.com>
To: Harald Alvestrand <harald@alvestrand.no>, "rtcweb@ietf.org" <rtcweb@ietf.org>
Thread-Topic: [rtcweb] Comments on H.264 and VP8 performance comparisons
Thread-Index: Ac7JIe76OhlqjGwCQFaenvACTkQ0VwF2Vc2AAA12J5D//+fngP//yUPg
Date: Tue, 22 Oct 2013 16:07:06 +0000
Message-ID: <BBE9739C2C302046BD34B42713A1E2A22DFC94DC@ESESSMB105.ericsson.se>
References: <BBE9739C2C302046BD34B42713A1E2A22DF9F8D2@ESESSMB105.ericsson.se> <52664A39.5040105@alvestrand.no> <BBE9739C2C302046BD34B42713A1E2A22DFC7383@ESESSMB105.ericsson.se> <52669059.4080700@alvestrand.no>
In-Reply-To: <52669059.4080700@alvestrand.no>
Accept-Language: sv-SE, en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [153.88.183.16]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrPLMWRmVeSWpSXmKPExsUyM+Jvje6aRWlBBq/38lgc6+tis1j7r53d gcnjyoQrrB5LlvxkCmCK4rJJSc3JLEst0rdL4Mr4tFu6YKJ9xatT+g2M1wy7GDk5JARMJDZu vcoIYYtJXLi3nq2LkYtDSOAoo0TnjclMEM4iRonuE1PZQKrYBDQk5u+4C9YhIhAs0fv8PZgt LOAu0XtrIlANB1DcQ6JtuQJEiZtE16UlYGEWAVWJlslhIGFeAV+Jjy/3sEOMv84osXjlZLDx nAK6EkuePAUbySggK3H/+z0WEJtZQFzi1pP5TBCHCkgs2XOeGcIWlXj5+B8rhK0osfNsOzNE vY7Egt2f2CBsbYllC18zQywWlDg58wnLBEbRWUjGzkLSMgtJyywkLQsYWVYxchSnFiflphsZ bGIERsLBLb8tdjBe/mtziFGag0VJnPfjW+cgIYH0xJLU7NTUgtSi+KLSnNTiQ4xMHJxSDYyF Gv+f/TQuF2m16Up9KnnD5vzBok9mymqJG7mb8zlyrZxeNXKH8nT1MpRIHtZfMCvpvJTqp3d9 OebBEkWia5YUFtxzlbCY49cftfLIkS3pHj76/WfafgZ0/mn3/LLb/JqyjuSWhcIcrz+brnfi 7Ppv//3M48hnnJq3O5klUvK1ZA0s3z9zUmIpzkg01GIuKk4EAKMEketSAgAA
Subject: Re: [rtcweb] Comments on H.264 and VP8 performance comparisons
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 22 Oct 2013 16:07:24 -0000

I can confirm we get 12.7% advantage for VP8 with the unmodified ada7d checkout.

> -----Original Message-----
> From: Harald Alvestrand [mailto:harald@alvestrand.no]
> Sent: den 22 oktober 2013 16:49
> To: Bo Burman; rtcweb@ietf.org
> Subject: Re: [rtcweb] Comments on H.264 and VP8 performance comparisons
> 
> On 10/22/2013 04:20 PM, Bo Burman wrote:
> > Hi all,
> >
> > We understand that there are many people that are interested in a VP8 vs x264 comparison that includes rate controls.
> >
> > We therefore performed a quick study of the VP8 and x264 rate controls to see whether there were any major
> differences between them in the way they select quantizers. We found that VP8 uses what can be described as QP-
> toggling: By lowering the QP of, say, every 6th frame, you increase the quality of these markedly. The following frames
> also benefit from the increased quality (even though they have to make do with less bits than before) since they can
> predict from the high-quality frame.
> >
> > However, we found that the rate-control of x264 does not QP-toggle in this fine granular way, and x264 may therefore
> be at a disadvantage when comparing against VP8. So we ran a test using a modified version of x264 to allow QP-toggling
> during rate control. The modification is very simple and a patch for the x264 code is found in the bottom of this email.
> >
> > With this change, H.264 baseline (implemented using the modified x264 with rate control turned on) performs equally
> to VP8 with rate-control on; the difference is 1.3% in favor of VP8, but if you swap the order of the two codecs in
> draw_graphs.sh, H.264 baseline wins with 1.6%. This result is thus a tie between VP8 and H.264 baseline, and this is
> consistent with our previous fixed-QP test, and far from the (at least) 13% advantage for VP8 stated by Google.
> >
> > The modification is simple and it may well be possible to improve x264 by more sophisticated modifications. But
> pursuing this track is what we oppose, we do not want comparisons where the outcome depends on how well you
> modify a particular rate control mechanism - the rate control is  codec-agnostic after all. To avoid this we still maintain our
> view that codec comparisons should be done using fixed QP settings, which is also the practice in MPEG and VCEG.
> >
> > We used the test parameters from Google's test repository
> https://git.chromium.org/git/webm/vpx_codec_comparison.git, the version
> ada7d7937a54e47fc18e2e0a1287aea29dc5d1c5.
> 
> Thanks!
> 
> Can you verify that you did see the same 12.7% difference when you ran the unmodified ada7d checkout? I want to be
> sure that I haven't missed any external influences on the results.
> 
> Having a 2.9% difference by swapping the order of arguments seems wrong; I'll pass this on to the guy who wrote that
> particular script.
> 
> >
> > Here are the x264 and comparison script patches:
> > --
> >
> > diff --git a/encoder/ratecontrol.c b/encoder/ratecontrol.c index
> > 1a39a49..2cf256f 100644
> > --- a/encoder/ratecontrol.c
> > +++ b/encoder/ratecontrol.c
> > @@ -1707,7 +1707,18 @@ int x264_ratecontrol_mb_qp( x264_t *h ) {
> >       x264_emms();
> >       float qp = h->rc->qpm;
> > -    if( h->param.rc.i_aq_mode )
> > +
> > +       if (h->sh.i_type != SLICE_TYPE_I){
> > +          if ((h->fenc->i_poc % 12 )== 0)
> > +              qp -= 3.0;
> > +          else
> > +              qp += 0.0;
> > +
> > +       } else {
> > +          qp -= 5.0;
> > +       }
> > +
> > +    if( h->param.rc.i_aq_mode)
> >       {
> >            /* MB-tree currently doesn't adjust quantizers in unreferenced frames. */
> >           float qp_offset = h->fdec->b_kept_as_ref ?
> > h->fenc->f_qp_offset[h->mb.i_mb_xy] :
> > h->fenc->f_qp_offset_aq[h->mb.i_mb_xy];
> >
> >
> > draw_graphs.sh:
> >
> > change
> > ./visual_metrics.py metrics_template.html "*0.txt" stats/h264
> > stats/vp8  > vp8_vs_h264_quality.html to ./visual_metrics.py
> > metrics_template.html "*0.txt" stats/vp8 stats/h264  >
> > vp8_vs_h264_quality2.html
> >
> > Best Regards,
> > Bo Burman
> >
> >> -----Original Message-----
> >> From: rtcweb-bounces@ietf.org [mailto:rtcweb-bounces@ietf.org] On
> >> Behalf Of Harald Alvestrand
> >> Sent: den 22 oktober 2013 11:50
> >> To: rtcweb@ietf.org
> >> Subject: Re: [rtcweb] Comments on H.264 and VP8 performance
> >> comparisons
> >>
> >> On 10/14/2013 11:12 PM, Bo Burman wrote:
> >>> Hi all,
> >>>
> >>> We would like to counter Google's suggestion that our test has only
> >>> "demonstrated that it is possible to reduce VP8's
> >> performance" (updated draft on VP8 http://datatracker.ietf.org/doc/draft-alvestrand-rtcweb-vp8/).
> >>> In fact, what we did in our test was mostly undoing some very
> >>> peculiar
> >>> x264 settings made by Google in their test from April 3. By instead
> >>> using the x264 settings Google themselves proposed in their earlier
> >>> test (from March 12), and removing threading, the difference went
> >>> down from 41% to 16%. (This is without touching the VP8 parameters.)
> >>>
> >>> The last change we made was to remove the rate control from the
> >>> comparison, something that is standard practice in
> >> the world of video standardization. This involved changing both the
> >> x264 and VP8 parameters. After that, the difference went down to -1%.
> >>> In summary, the following steps were taken in our comparison:
> >>>
> >>> 1) Downloading the latest software: 41% became 36%
> >>> 2) Removing threading: 26%
> >>> 3) Removing bit padding: 18%
> >>> 4) Removing other differences between Google's March 12 and April
> >>> 3rd
> >>> tests: 16%
> >>> 5) Removing rate controller: -1%
> >>>
> >>>
> >> Just a quick update on this - I did not manage to get a new draft
> >> ready before the deadline, so I'll have to resort to sending email:
> >>
> >> I applied steps 1), 2) and 3) to the repository mentioned in the draft.
> >> The numbers I got were different, but significant. Below are the
> >> VP8-to-x264 differences I encountered each step of the way.
> >>
> >> - Master branch before October 15: Difference 71.52%
> >> - Updating x264 from 198a7ea (aug 16 2012) to c832fe (March 1 2013):
> >> Difference 71.44%
> >>     (I could not go beyond this because yasm 1.2 was required for newer versions).
> >> - Adding the --thread 1 parameter: Difference 26.11%
> >> - Removing the --nai-hrd=cbr parameter that was suggested by x264
> >> people: Difference 13.87%
> >> - Removing vbv-maxrate and vbv-init 0.8: Difference 12.74%
> >>
> >> I did not shorten the clips to 10 seconds, nor did I try to control
> >> rate by constant cq instead of a rate controller; if Bo's numbers are right, this shouldn't matter.
> >>
> >> I did try removing "vbv-bufsize ${rate}", since this was the
> >> remaining difference I could find for Bo's point 4 "other differences", but that was actually harmful (difference
> increased to 19.05%), so I put it back.
> >>
> >> I checked this in to the repository - the commit is here:
> >>
> >> http://git.chromium.org/gitweb/?p=webm/vpx_codec_comparison.git;a=com
> >> mitdiff;h=ada7d7937a54e47fc18e2e0a1287
> >> aea29dc5d1c5
> >>
> >>
> >> This number does not fit the impression the video codec team has from
> >> other tests - they think
> >> VP8 can do a lot more than 13% better than baseline - but at the
> >> draft deadline, we had not found the set of VP8 parameters that showed this for this particular clip set.
> >>
> >> Commentary: I'm surprised at x264's choice of default value for the
> >> --thread parameter. Accepting a 26% bitrate hit seems like a large price to pay for going faster by default. Is there a
> bug here?
> >>
> >> _______________________________________________
> >> rtcweb mailing list
> >> rtcweb@ietf.org
> >> https://www.ietf.org/mailman/listinfo/rtcweb