Re: [rtcweb] More H.264 vs VP8 tests

Stefan Håkansson LK <stefan.lk.hakansson@ericsson.com> Tue, 25 June 2013 14:59 UTC

Return-Path: <stefan.lk.hakansson@ericsson.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 344C121F9B3E for <rtcweb@ietfa.amsl.com>; Tue, 25 Jun 2013 07:59:42 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.428
X-Spam-Level:
X-Spam-Status: No, score=-5.428 tagged_above=-999 required=5 tests=[AWL=0.521, BAYES_00=-2.599, HELO_EQ_SE=0.35, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id svZ2myW1CPZ7 for <rtcweb@ietfa.amsl.com>; Tue, 25 Jun 2013 07:59:36 -0700 (PDT)
Received: from mailgw7.ericsson.se (mailgw7.ericsson.se [193.180.251.48]) by ietfa.amsl.com (Postfix) with ESMTP id D701211E810F for <rtcweb@ietf.org>; Tue, 25 Jun 2013 07:59:31 -0700 (PDT)
X-AuditID: c1b4fb30-b7f7d6d000001cf9-3c-51c9b0529e2e
Received: from ESESSHC022.ericsson.se (Unknown_Domain [153.88.253.125]) by mailgw7.ericsson.se (Symantec Mail Security) with SMTP id 45.F2.07417.250B9C15; Tue, 25 Jun 2013 16:59:30 +0200 (CEST)
Received: from ESESSMB209.ericsson.se ([169.254.9.6]) by ESESSHC022.ericsson.se ([153.88.183.84]) with mapi id 14.02.0328.009; Tue, 25 Jun 2013 16:59:30 +0200
From: =?iso-8859-1?Q?Stefan_H=E5kansson_LK?= <stefan.lk.hakansson@ericsson.com>
To: Harald Alvestrand <harald@alvestrand.no>
Thread-Topic: [rtcweb] More H.264 vs VP8 tests
Thread-Index: Ac5vQ4bE4hnmu5ERSv6YOg93i7vBVw==
Date: Tue, 25 Jun 2013 14:59:30 +0000
Message-ID: <1447FA0C20ED5147A1AA0EF02890A64B1C302C4E@ESESSMB209.ericsson.se>
References: <BBE9739C2C302046BD34B42713A1E2A22DECC12F@ESESSMB105.ericsson.se> <51C96E36.2000907@alvestrand.no>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [153.88.183.16]
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrILMWRmVeSWpSXmKPExsUyM+JvrW7QhpOBBlNPKlsc6+tis1j7r53d gcnjyoQrrB5LlvxkCmCK4rJJSc3JLEst0rdL4Mq48WIFU8EBlYqby5rZGxivynYxcnJICJhI TDt9hxnCFpO4cG89WxcjF4eQwGFGiUdT9kM5ixgllnR+ZwOpYhMIlNi6bwGYLSKgI/FwfwMT iM0s4C2xvnsOC4gtLKAr0XB8DVSNnsTC5y9YYOzd+z+CbWMRUJU4s6sLrIZXwFfi/Yk2VhBb SKBAYuqmLWAzGYEu+n5qDdR8cYlbT+YzQVwqILFkz3moq0UlXj7+xwphK0rsPNvODFGvJ3Fj 6hQ2CFtbYtnC18wQuwQlTs58wjKBUXQWkrGzkLTMQtIyC0nLAkaWVYzsuYmZOenl5psYgdFw cMtvgx2Mm+6LHWKU5mBREuf9dGpXoJBAemJJanZqakFqUXxRaU5q8SFGJg5OqQbGSX+Utmxj 4S1a7/tL2uXT/JrJugabD3yW6Fj3zkWMuS801vHViS8S8v1m29NTF3eLqPxterCL6dE57U1c Aotqah1EP/69kDK5NvbI/G23Z66+2PJu7ZPalIV2xTN3H0k8dmjn6aqwF5VJZrLpDOmChi+V WX40q95a1Na8tZ7F//Sa0qj5Z/QPKrEUZyQaajEXFScCAEJJRTBUAgAA
Cc: "rtcweb@ietf.org" <rtcweb@ietf.org>
Subject: Re: [rtcweb] More H.264 vs VP8 tests
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 25 Jun 2013 14:59:46 -0000

On 2013-06-25 12:17, Harald Alvestrand wrote:
> Again - thanks for releasing this openly!
>
> I ran the scripts (with a few tweaks; you run on a system where sh is
> bash, not dash, for instance), and got the same numbers within +/- 0.5%
> (probably some binary version skew); we may have disagreements on the
> parameters to use, but we agree on the numbers those parameters produce.

Great. We spent quite some time trying to find the best parameter 
settings, but it is quite complex and we may have missed something - 
feedback is welcome.

>
> (I have since modified the Google framework to include a script that
> pulls in the sources for the needed binaries and compiles them - if you
> want to make 100% sure people are working from the same sources, you may
> want to rebase to a newer version of the comparision toolkit.)

We'll look into that.

>
> On 06/22/2013 03:41 PM, Bo Burman wrote:
>> Hi all,
>>
>> We have had a look at Google's comparison between VP8 and H.264 constrained baseline that was posted on April 3rd (http://www.ietf.org/mail-archive/web/rtcweb/current/msg07028.html). This post contains, as the one mentioned above (and if the attachments make it to the list), information on the exact tools and options used for encoding and should thus be repeatable by anyone interested.
>>
>> As was already stated by others on this list, one major problem is that Google's test involves the rate control mechanism. Typically codecs are measured with rate control turned off, since it acts as a huge noise on the measurement. Instead we propose to compare the codecs using fixed qp-levels. The qp-level is the quantization parameter that affects the rate/distortion tradeoff. Comparing using fixed qp-levels is what has been used when benchmarking HEVC against H.264 in the JCT-VC standardization, for instance. We are going to select a codec (essentially bit stream format), not a rate control mechanism: Once the codec is selected you can choose whatever rate control mechanism you wish.
>>
>> We used Google's excellent framework as the baseline and changed the parameter settings in order to make it possible to measure using fixed qp. We used the same sequences, but limited them to the first 10 seconds since they varied from 10 seconds to minutes; this also eased computation time.
>>
>> We used two H.264 encoder implementations: X264, which is an open-source codec that can operate in everything from real-time to slow, and JM which is the reference implementation that was used to develop H.264. JM is very slow but attempts to be very efficient in terms of bits per quality. The results were as follows:
>>
>> X264 baseline vs VP8: H.264 wins with 1%
>> JM baseline vs VP8: H.264 wins with 4%
>>
>> Running times:
>> X264: 1 hour 3 minutes
>> VP8: 2 hours 0 minutes
>> JM: order of magnitude slower
>>
>> It is interesting to note that the measurements are more stable in the new test; the variance of the percentages for the sequences is now around 70, down from around 700 in Google's test of April 3rd.  We believe this is due to the removal of the rate controller, which acts like noise on the measurements.
>>
>> We also tried setting H.264 to constrained high (no interlace and no B-pictures, compared to high). The results were then:
>>
>> X264 constrained high vs VP8: H.264 wins with 25%
>> JM constrained high vs VP8: H.264 wins with 24%
>>
>> We also note that the script that Google provided to calculate the rate differences ("BD-rate") does not give exactly the same numbers as the JCT-VC-way of calculating BD-rate. The main difference is that the JM score for constrained high is better (around 29%) if the JCT-VC way of calculating BD-rate is used.
>>
>> In summary we think that proper testing can conclude that there is no clear performance advantage to any codec between VP8 and H.264 baseline. When comparing VP8 against H.264 constrained high on the other hand, it seems like there is an advantage for H.264 constrained high.
>>
>> The attached file includes the files necessary to reproduce the test.
>>
>> Best Regards,
>>
>> Bo Burman
>>
>>
>> _______________________________________________
>> rtcweb mailing list
>> rtcweb@ietf.org
>> https://www.ietf.org/mailman/listinfo/rtcweb
>