From: Bo Burman <bo.burman@ericsson.com>
To: "rtcweb@ietf.org" <rtcweb@ietf.org>
Thread-Topic: Comments on H.264 and VP8 performance comparisons
Thread-Index: Ac7JIe76OhlqjGwCQFaenvACTkQ0Vw==
Date: Mon, 14 Oct 2013 21:12:08 +0000
Message-ID: <BBE9739C2C302046BD34B42713A1E2A22DF9F8D2@ESESSMB105.ericsson.se>
Accept-Language: sv-SE, en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Subject: [rtcweb] Comments on H.264 and VP8 performance comparisons
Precedence: list

Hi all,

We would like to counter Google's suggestion that our test has only "demons=
trated that it is possible to reduce VP8's performance" (updated draft on V=
P8 http://datatracker.ietf.org/doc/draft-alvestrand-rtcweb-vp8/).=20

In fact, what we did in our test was mostly undoing some very peculiar x264=
 settings made by Google in their test from April 3. By instead using the x=
264 settings Google themselves proposed in their earlier test (from March 1=
2), and removing threading, the difference went down from 41% to 16%. (This=
 is without touching the VP8 parameters.)

The last change we made was to remove the rate control from the comparison,=
 something that is standard practice in the world of video standardization.=
 This involved changing both the x264 and VP8 parameters. After that, the d=
ifference went down to -1%.

In summary, the following steps were taken in our comparison:

1) Downloading the latest software: 41% became 36%
2) Removing threading: 26%
3) Removing bit padding: 18%
4) Removing other differences between Google's March 12 and April 3rd tests=
: 16%
5) Removing rate controller: -1%

Contrary to Google's note on our test the purpose was not to "reduce the VP=
8 performance" but rather to present a technically correct codec comparison=
. Below follows a detailed description of what we found:

---

When we first saw Google's test which was posted on April 3rd, we were surp=
rised to find that their results differed so much from our own. Whereas we =
got a negligible difference between VP8 and H.264 constrained baseline, Goo=
gle reported that H.264 constrained baseline needed 41% more bits than VP8 =
for the same quality. This made us look deeper into Google's test to see ho=
w this could be explained.=20

The first thing we did was to download the latest versions of all software =
packages. By using the newest version of x264, the difference went down fro=
m 41% to 36%.

The second thing that caught our attention was that Google's x264 test used=
 auto threading. By omitting the parameter "--threads 1", the x264 encoder =
defaults to "--threads auto", which means that a large number of threads wi=
ll be used to compress the image (see for instance http://mewiki.project357=
.com/wiki/X264_Settings). This will have the effect of decreasing the compr=
ession time at the cost of quality. The VP8 codec, on the other hand, defau=
lts to a single thread with no quality degradation. Even when we changed th=
e x264 configuration to use the proper threading value ("--threads 1"), the=
 x264 codec was twice as fast as VP8, and the difference in bit rate went d=
own from 36% to 26%.=20

At this time we proceeded to only test the first 10 seconds of each sequenc=
e in order to get reasonable running times. This actually increased the dif=
ference again to 29%.

Our attention now turned to the "--nal-hrd cbr" parameter in Google's x264 =
April 3rd parameter list. As described at http://mewiki.project357.com/wiki=
/X264_Settings#nal-hrd, this setting will pack the bitstream with padding b=
its in order to exactly reach a particular bit rate. This is useful in some=
 circumstances such as in Blu-ray or ISDN video telephony which has to be e=
xactly, say, 64000 bits per second, but it is undesirable in a codec compar=
ison since it will only add bits and not increase quality. The VP8 encoder =
does not do such bit padding and was thus at an advantage. Removing the "--=
nal-hrd cbr" parameter for x264 avoided the unnecessary bit padding and the=
 difference now went down from 29% to 18%.

At this point in time we tried removing the remaining variables that differ=
ed between the x264 parameter set of Google's first test (from March 12) an=
d the x264 parameter set of Google's second test (from April 3rd), and this=
 resulted in a further decrease from 18% to 16%.

Finally we removed the rate controller from the test and instead used fixed=
 QP. As we have argued previously, having a rate controller in the loop jus=
t adds noise to the test and also risks measuring the performance of the ra=
te controller rather than the codec. Using fixed QP (no rate control) is th=
erefore established practice in the video codec community. As an example, t=
he Motion Pictures Expert Group (MPEG) recommends using fixed QP for video =
comparisons, see for instance (http://mpeg.chiariglione.org/standards/explo=
ration/internet-video-coding/call-proposals-internet-video-coding-technolog=
y). The reworked test without rate control (which we published on June 22) =
then got the result of -1%. This means H.264 constrained baseline (in the x=
264 implementation) outperformed VP8. We also found H.264 constrained high,=
 again using the x264 implementation, to be 24% better than VP8.

Google's comparison on April 3rd also included a speed test. In this, x264 =
is set to use only one thread, the parameter "--threads 1" is set. This mea=
ns that x264 cannot enjoy the faster speed of a parallel implementation. We=
 do not quite understand this choice of parameters from Google: When thread=
ing should be avoided (in the case of measuring quality), it is used, where=
as when it would be helpful (in the case of measuring speed), it is avoided=
. In both cases this is unfavorable to x264. This does not seem entirely fa=
ir.

Note that the biggest differences in performance were not due the changes m=
ade to the VP8 settings but to those of x264: The threading and the bit pad=
ding alone accounted for 21 of the 41 percentage points. Thus, we do not th=
ink that our test is about trying to "reduce the VP8 performance". Instead,=
 the major things have been avoiding what we believe are unfortunate parame=
ter choices for x264 on behalf of Google, and to create a technically corre=
ct test.

Best Regards,

Bo Burman