Re: [rtcweb] More H.264 vs VP8 tests

Stefan Håkansson LK <stefan.lk.hakansson@ericsson.com> Wed, 26 June 2013 21:24 UTC

Return-Path: <stefan.lk.hakansson@ericsson.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8AA9521F9B81 for <rtcweb@ietfa.amsl.com>; Wed, 26 Jun 2013 14:24:29 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.124
X-Spam-Level:
X-Spam-Status: No, score=-4.124 tagged_above=-999 required=5 tests=[AWL=-1.825, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1Rzyy1ph8P5S for <rtcweb@ietfa.amsl.com>; Wed, 26 Jun 2013 14:24:24 -0700 (PDT)
Received: from sesbmg20.ericsson.net (sesbmg20.ericsson.net [193.180.251.56]) by ietfa.amsl.com (Postfix) with ESMTP id 3E37421F9ABD for <rtcweb@ietf.org>; Wed, 26 Jun 2013 14:24:22 -0700 (PDT)
X-AuditID: c1b4fb38-b7f176d000001730-fb-51cb5c01291c
Received: from ESESSHC001.ericsson.se (Unknown_Domain [153.88.253.125]) by sesbmg20.ericsson.net (Symantec Mail Security) with SMTP id 99.10.05936.10C5BC15; Wed, 26 Jun 2013 23:24:17 +0200 (CEST)
Received: from ESESSMB209.ericsson.se ([169.254.9.6]) by ESESSHC001.ericsson.se ([153.88.183.21]) with mapi id 14.02.0328.009; Wed, 26 Jun 2013 23:24:17 +0200
From: =?iso-8859-1?Q?Stefan_H=E5kansson_LK?= <stefan.lk.hakansson@ericsson.com>
To: John Koleszar <jkoleszar@google.com>
Thread-Topic: [rtcweb] More H.264 vs VP8 tests
Thread-Index: Ac5vQ4bE4hnmu5ERSv6YOg93i7vBVw==
Date: Wed, 26 Jun 2013 21:24:16 +0000
Message-ID: <1447FA0C20ED5147A1AA0EF02890A64B1C3055CA@ESESSMB209.ericsson.se>
References: <BBE9739C2C302046BD34B42713A1E2A22DECC12F@ESESSMB105.ericsson.se> <CAAvHm8P5w_sy_cRScudpg7RytxZJRhxykmHtw1dFMB42BYF5-g@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [153.88.183.148]
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrALMWRmVeSWpSXmKPExsUyM+JvrS5jzOlAg7ePtCx+r9zIarH2Xzu7 A5PHgk2lHkuW/GQKYIrisklJzcksSy3St0vgyph35iJLwVGlimcrJ7A1MG6Q6WLk5JAQMJHY 9X0nI4QtJnHh3nq2LkYuDiGBo4wSf36eYQVJCAksYpR4cVoPxGYTCJTYum8BG4gtIqAh0bF9 IjOIzSzgLbG+ew4LiC0soCvRcHwNVI2exMLnL1hg7N37PwLVc3CwCKhKXL3CAxLmFfCV+D5r PTPE3imMEj1nHoL1MgId9P3UGiaI+eISt57MZ4I4VEBiyZ7zzBC2qMTLx/9YIWwlicYlT1gh 6vUkbkydwgZha0ssW/iaGWKZoMTJmU9YJjCKzkIydhaSlllIWmYhaVnAyLKKkaM4tTgpN93I YBMjMBYObvltsYPx8l+bQ4zSHCxK4ryfTu0KFBJITyxJzU5NLUgtii8qzUktPsTIxMEp1cCY Y9tz8cGuh0vqzCcERYaFXVqZWtQpf4ib82VM7dxmfZ/E7RNnv7n7evPSzKXamXavjXfcLuNP vlJlYcQjrODDvXnrhM17FwUyTBCKFJ+5WeV8Ycj+V/Ok3RZOPaWUsbX9ypVn6qYPdorLtbDU ZYh9ux2qeLi0ofijxoN8qQxj3Zw7mQknjacosRRnJBpqMRcVJwIAvfP6VlMCAAA=
Cc: "rtcweb@ietf.org" <rtcweb@ietf.org>
Subject: Re: [rtcweb] More H.264 vs VP8 tests
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 26 Jun 2013 21:24:30 -0000

On 2013-06-25 17:36, John Koleszar wrote:
> On Sat, Jun 22, 2013 at 6:41 AM, Bo Burman <bo.burman@ericsson.com>
> wrote:
>> Hi all,
>>
>> We have had a look at Google's comparison between VP8 and H.264
>> constrained baseline that was posted on April 3rd
>> (http://www.ietf.org/mail-archive/web/rtcweb/current/msg07028.html).
>> This post contains, as the one mentioned above (and if the
>> attachments make it to the list), information on the exact tools
>> and options used for encoding and should thus be repeatable by
>> anyone interested.
>>
>> As was already stated by others on this list, one major problem is
>> that Google's test involves the rate control mechanism. Typically
>> codecs are measured with rate control turned off, since it acts as
>> a huge noise on the measurement. Instead we propose to compare the
>> codecs using fixed qp-levels. The qp-level is the quantization
>> parameter that affects the rate/distortion tradeoff. Comparing
>> using fixed qp-levels is what has been used when benchmarking HEVC
>> against H.264 in the JCT-VC standardization, for instance. We are
>> going to select a codec (essentially bit stream format), not a rate
>> control mechanism: Once the codec is selected you can choose
>> whatever rate control mechanism you wish.
>>
>> We used Google's excellent framework as the baseline and changed
>> the parameter settings in order to make it possible to measure
>> using fixed qp. We used the same sequences, but limited them to the
>> first 10 seconds since they varied from 10 seconds to minutes; this
>> also eased computation time.
>>
>> We used two H.264 encoder implementations: X264, which is an
>> open-source codec that can operate in everything from real-time to
>> slow, and JM which is the reference implementation that was used to
>> develop H.264. JM is very slow but attempts to be very efficient in
>> terms of bits per quality. The results were as follows:
>>
>> X264 baseline vs VP8: H.264 wins with 1% JM baseline vs VP8: H.264
>> wins with 4%
>>
>> Running times: X264: 1 hour 3 minutes VP8: 2 hours 0 minutes JM:
>> order of magnitude slower
>>
>> It is interesting to note that the measurements are more stable in
>> the new test; the variance of the percentages for the sequences is
>> now around 70, down from around 700 in Google's test of April 3rd.
>> We believe this is due to the removal of the rate controller, which
>> acts like noise on the measurements.
>>
>> We also tried setting H.264 to constrained high (no interlace and
>> no B-pictures, compared to high). The results were then:
>>
>> X264 constrained high vs VP8: H.264 wins with 25% JM constrained
>> high vs VP8: H.264 wins with 24%
>>
> [...]
>
> At one point, running the libvpx implementation of VP8 in fixed qp
> mode would effectively limit the encoder to only a single reference
> frame. I don't recall if it's still the case.

We have stepped through the decoding of the .webm files produced using 
our parameter settings and they indeed use several reference frames, so 
no, this seems not to be the case.

> Another unexpected
> behavior for people trying this mode is that making libvpx bump up
> against the lower quantizer without setting an approximately correct
> target bitrate can also put the encoder into its "over quant" mode,
> where it'll throw away additional residual trying to hit the target
> bitrate.

We have not seen this happen either. The target bitrate we set is 
ridiculously high so we do not see why the encoder should throw away 
anything that would cost bits, rather it would do the opposite.

> It doesn't have anything like the x264 ipratio (though I
> see you set this to 1, but this is something that bites people
> trying fixed-qp). This mode isn't really supported by libvpx, so I'd
> be careful about taking these results as representative of the very
> best VP8 can do. _______________________________________________
> rtcweb mailing list rtcweb@ietf.org
> https://www.ietf.org/mailman/listinfo/rtcweb
>