Re: [rtcweb] More H.264 vs VP8 tests

Stefan Håkansson LK <stefan.lk.hakansson@ericsson.com> Tue, 02 July 2013 10:16 UTC

Return-Path: <stefan.lk.hakansson@ericsson.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2B82E11E846B for <rtcweb@ietfa.amsl.com>; Tue, 2 Jul 2013 03:16:12 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.82
X-Spam-Level:
X-Spam-Status: No, score=-3.82 tagged_above=-999 required=5 tests=[AWL=-1.521, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IIjTClbgBAY4 for <rtcweb@ietfa.amsl.com>; Tue, 2 Jul 2013 03:16:07 -0700 (PDT)
Received: from sesbmg20.ericsson.net (sesbmg20.ericsson.net [193.180.251.56]) by ietfa.amsl.com (Postfix) with ESMTP id D81E911E8476 for <rtcweb@ietf.org>; Tue, 2 Jul 2013 03:16:04 -0700 (PDT)
X-AuditID: c1b4fb38-b7f456d000002e83-5e-51d2a86203e2
Received: from ESESSHC022.ericsson.se (Unknown_Domain [153.88.253.125]) by sesbmg20.ericsson.net (Symantec Mail Security) with SMTP id 7F.F5.11907.268A2D15; Tue, 2 Jul 2013 12:16:03 +0200 (CEST)
Received: from ESESSMB209.ericsson.se ([169.254.9.6]) by ESESSHC022.ericsson.se ([153.88.183.84]) with mapi id 14.02.0328.009; Tue, 2 Jul 2013 12:16:02 +0200
From: =?iso-8859-1?Q?Stefan_H=E5kansson_LK?= <stefan.lk.hakansson@ericsson.com>
To: Saverio Mascolo <saverio.mascolo@gmail.com>
Thread-Topic: [rtcweb] More H.264 vs VP8 tests
Thread-Index: Ac5vQ4bE4hnmu5ERSv6YOg93i7vBVw==
Date: Tue, 2 Jul 2013 10:16:02 +0000
Message-ID: <1447FA0C20ED5147A1AA0EF02890A64B1C309A30@ESESSMB209.ericsson.se>
References: <BBE9739C2C302046BD34B42713A1E2A22DECC12F@ESESSMB105.ericsson.se> <51C96E36.2000907@alvestrand.no> <1447FA0C20ED5147A1AA0EF02890A64B1C308D3B@ESESSMB209.ericsson.se> <CAK1jYfdDRCJjGNwU1nimmKEqThSQSn9=7zEJ3PJXdru+MAT1fQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [153.88.183.146]
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsUyM+JvrW7yikuBBrt3MFoc6+tis1j7r53d 4vi/0+wOzB5XJlxh9dg56y67x5IlP5kCmKO4bFJSczLLUov07RK4Mv7M+sFcMMuq4tmuRYwN jD06XYycHBICJhLT399jg7DFJC7cWw9mCwkcZZSY/s6ki5ELyF7IKPFv5nVWkASbQKDE1n0L gIo4OEQE9CXebUwGCTMLBEv0dk0GKxEW0JVoOL4GbI6IgJ7EwucvWGDs3fs/MoPYLAIqErOO /AOr5xXwlfjecoodYtdvRombj3azgyQYgQ76fmoNE8QCcYlbT+YzQRwqILFkz3lmCFtU4uVj iEESAkoSPzZcYoGo15O4MXUKG4StLbFs4WtmiGWCEidnPmGZwCg6C8nYWUhaZiFpmYWkZQEj yypGjuLU4qTcdCODTYzACDm45bfFDsbLf20OMUpzsCiJ827ROxMoJJCeWJKanZpakFoUX1Sa k1p8iJGJg1OqgXFeV9lN9nTpVbVPOdX8Fuw6Kqz5rr7Lp+//AbtHgR8dv6wrzdVtWtcxJ1Gj oFLmitCRzs/vJ1pKT3j+4cf0VYc05wfcmbZHZ1lskrnn3NSrW+df2t5xPTR+zT6F8+cLvu/J nRm+j4ffoPfKxTSfB/9rvsxe9XrhiY/hZnsY/bkyLmatef1hsfx1JZbijERDLeai4kQAPIes 3V4CAAA=
Cc: "rtcweb@ietf.org" <rtcweb@ietf.org>
Subject: Re: [rtcweb] More H.264 vs VP8 tests
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 02 Jul 2013 10:16:12 -0000

On 2013-07-01 19:10, Saverio Mascolo wrote:
> an evaluation of how h264 and vp8 are able to match a changing quality
> would be interesting

Such an evaluation would be very hard to do. The H.264 standard does not 
specify any encoder behavior, only the decoder is specified. This means 
that the encoder can be implemented freely, as long as it produces a 
bitstream compliant to the standard. Specifically, there is no 
specification in H.264 on how to implement the rate controller. Thus the 
rate control mechanisms of two different H.264 encoder implementations 
could react differently to a change in scene complexity.

There is a wealth of H.264 implementations on the market today. Some may 
have implemented a better rate control mechanism than x264, some may 
have implemented a worse variant. Testing x264 versus the WebM vp8 
implemenation with rate control turned on may give some information 
about the relative merits of these two rate control implementations 
(although the rate control gives rise to so much measurement noise so 
even that is hard to do) but it says very little about how the H.264 
standard compares to the vp8 specification in situations where the scene 
complexity varies.


>
>
> On Sat, Jun 29, 2013 at 3:17 PM, Stefan Håkansson LK
> <stefan.lk.hakansson@ericsson.com
> <mailto:stefan.lk.hakansson@ericsson.com>> wrote:
>
>     On 6/25/13 12:17 PM, Harald Alvestrand wrote:
>      > Again - thanks for releasing this openly!
>      >
>      > I ran the scripts (with a few tweaks; you run on a system where sh is
>      > bash, not dash, for instance), and got the same numbers within
>     +/- 0.5%
>      > (probably some binary version skew);
>
>     I re-run the scripts on another computer with another OS today, and I
>     get exactly the same results as Bo sent out. I noted however that if the
>     input clips are not cut at 10s (but used in their entire length) the
>     results get slightly different, but within +/- 0.5%. Can this be the
>     reason why you get slightly different numbers?
>
>
>      > we may have disagreements on the
>      > parameters to use, but we agree on the numbers those parameters
>     produce.
>      >
>      > (I have since modified the Google framework to include a script that
>      > pulls in the sources for the needed binaries and compiles them -
>     if you
>      > want to make 100% sure people are working from the same sources,
>     you may
>      > want to rebase to a newer version of the comparision toolkit.)
>      >
>      > On 06/22/2013 03:41 PM, Bo Burman wrote:
>      >> Hi all,
>      >>
>      >> We have had a look at Google's comparison between VP8 and H.264
>     constrained baseline that was posted on April 3rd
>     (http://www.ietf.org/mail-archive/web/rtcweb/current/msg07028.html).
>     This post contains, as the one mentioned above (and if the
>     attachments make it to the list), information on the exact tools and
>     options used for encoding and should thus be repeatable by anyone
>     interested.
>      >>
>      >> As was already stated by others on this list, one major problem
>     is that Google's test involves the rate control mechanism. Typically
>     codecs are measured with rate control turned off, since it acts as a
>     huge noise on the measurement. Instead we propose to compare the
>     codecs using fixed qp-levels. The qp-level is the quantization
>     parameter that affects the rate/distortion tradeoff. Comparing using
>     fixed qp-levels is what has been used when benchmarking HEVC against
>     H.264 in the JCT-VC standardization, for instance. We are going to
>     select a codec (essentially bit stream format), not a rate control
>     mechanism: Once the codec is selected you can choose whatever rate
>     control mechanism you wish.
>      >>
>      >> We used Google's excellent framework as the baseline and changed
>     the parameter settings in order to make it possible to measure using
>     fixed qp. We used the same sequences, but limited them to the first
>     10 seconds since they varied from 10 seconds to minutes; this also
>     eased computation time.
>      >>
>      >> We used two H.264 encoder implementations: X264, which is an
>     open-source codec that can operate in everything from real-time to
>     slow, and JM which is the reference implementation that was used to
>     develop H.264. JM is very slow but attempts to be very efficient in
>     terms of bits per quality. The results were as follows:
>      >>
>      >> X264 baseline vs VP8: H.264 wins with 1%
>      >> JM baseline vs VP8: H.264 wins with 4%
>      >>
>      >> Running times:
>      >> X264: 1 hour 3 minutes
>      >> VP8: 2 hours 0 minutes
>      >> JM: order of magnitude slower
>      >>
>      >> It is interesting to note that the measurements are more stable
>     in the new test; the variance of the percentages for the sequences
>     is now around 70, down from around 700 in Google's test of April
>     3rd.  We believe this is due to the removal of the rate controller,
>     which acts like noise on the measurements.
>      >>
>      >> We also tried setting H.264 to constrained high (no interlace
>     and no B-pictures, compared to high). The results were then:
>      >>
>      >> X264 constrained high vs VP8: H.264 wins with 25%
>      >> JM constrained high vs VP8: H.264 wins with 24%
>      >>
>      >> We also note that the script that Google provided to calculate
>     the rate differences ("BD-rate") does not give exactly the same
>     numbers as the JCT-VC-way of calculating BD-rate. The main
>     difference is that the JM score for constrained high is better
>     (around 29%) if the JCT-VC way of calculating BD-rate is used.
>      >>
>      >> In summary we think that proper testing can conclude that there
>     is no clear performance advantage to any codec between VP8 and H.264
>     baseline. When comparing VP8 against H.264 constrained high on the
>     other hand, it seems like there is an advantage for H.264
>     constrained high.
>      >>
>      >> The attached file includes the files necessary to reproduce the
>     test.
>      >>
>      >> Best Regards,
>      >>
>      >> Bo Burman
>      >>
>      >>
>      >> _______________________________________________
>      >> rtcweb mailing list
>      >> rtcweb@ietf.org <mailto:rtcweb@ietf.org>
>      >> https://www.ietf.org/mailman/listinfo/rtcweb
>      >
>
>     _______________________________________________
>     rtcweb mailing list
>     rtcweb@ietf.org <mailto:rtcweb@ietf.org>
>     https://www.ietf.org/mailman/listinfo/rtcweb
>
>
>