Re: [rtcweb] More H.264 vs VP8 tests

Saverio Mascolo <saverio.mascolo@gmail.com> Mon, 01 July 2013 17:10 UTC

Return-Path: <saverio.mascolo@gmail.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 643C611E8208 for <rtcweb@ietfa.amsl.com>; Mon, 1 Jul 2013 10:10:09 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.299
X-Spam-Level:
X-Spam-Status: No, score=-2.299 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_8BIT_HEADER=0.3, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NmXQGIQC5uiD for <rtcweb@ietfa.amsl.com>; Mon, 1 Jul 2013 10:10:08 -0700 (PDT)
Received: from mail-ie0-x233.google.com (mail-ie0-x233.google.com [IPv6:2607:f8b0:4001:c03::233]) by ietfa.amsl.com (Postfix) with ESMTP id 5F52311E8232 for <rtcweb@ietf.org>; Mon, 1 Jul 2013 10:10:08 -0700 (PDT)
Received: by mail-ie0-f179.google.com with SMTP id c10so9833655ieb.38 for <rtcweb@ietf.org>; Mon, 01 Jul 2013 10:10:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=hKw+1HgjZmytbOMI1NASpRLo0rjsZ21PBYfqvPMn9KU=; b=Pozt72HWPJ5I0KPT3QeZcQc5zviow5FAtx4OFAGp1f9/CRyQnv4kkuMiJgwYELdoof EmZD6/Glvsoy038xBOeNtEONH414iV5p6Gp0q+5Gif1BrE2mzkKdc1rubRJULcUV+iy5 6PKRNEJtsninCXUyhvIvUDYQcv21m7i2D+Eju+bU6B9a2a1KxRX2NGFo9LR35UpsA7C/ VRCH2GX3nL1R1HnDC8WEDyAVbmCtUv6zav4aA/frRQiclOOcmVovjELoJ83tBR40W7qM 0VFPeGzQ1Ovg3zGlxuLuRCK7d8UTRgQA3xxvPw53Z8uLcKBaQjvSxGl5SsnqywGnguG/ IEPA==
X-Received: by 10.50.122.100 with SMTP id lr4mr16563829igb.18.1372698607854; Mon, 01 Jul 2013 10:10:07 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.64.138.166 with HTTP; Mon, 1 Jul 2013 10:09:47 -0700 (PDT)
In-Reply-To: <1447FA0C20ED5147A1AA0EF02890A64B1C308D3B@ESESSMB209.ericsson.se>
References: <BBE9739C2C302046BD34B42713A1E2A22DECC12F@ESESSMB105.ericsson.se> <51C96E36.2000907@alvestrand.no> <1447FA0C20ED5147A1AA0EF02890A64B1C308D3B@ESESSMB209.ericsson.se>
From: Saverio Mascolo <saverio.mascolo@gmail.com>
Date: Mon, 1 Jul 2013 19:09:47 +0200
Message-ID: <CAK1jYfdDRCJjGNwU1nimmKEqThSQSn9=7zEJ3PJXdru+MAT1fQ@mail.gmail.com>
To: =?ISO-8859-1?Q?Stefan_H=E5kansson_LK?= <stefan.lk.hakansson@ericsson.com>
Content-Type: multipart/alternative; boundary=001a11c1f7a4e7aa6a04e0764cad
Cc: "rtcweb@ietf.org" <rtcweb@ietf.org>
Subject: Re: [rtcweb] More H.264 vs VP8 tests
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 01 Jul 2013 17:10:09 -0000

an evaluation of how h264 and vp8 are able to match a changing quality
would be interesting


On Sat, Jun 29, 2013 at 3:17 PM, Stefan Håkansson LK <
stefan.lk.hakansson@ericsson.com> wrote:

> On 6/25/13 12:17 PM, Harald Alvestrand wrote:
> > Again - thanks for releasing this openly!
> >
> > I ran the scripts (with a few tweaks; you run on a system where sh is
> > bash, not dash, for instance), and got the same numbers within +/- 0.5%
> > (probably some binary version skew);
>
> I re-run the scripts on another computer with another OS today, and I
> get exactly the same results as Bo sent out. I noted however that if the
> input clips are not cut at 10s (but used in their entire length) the
> results get slightly different, but within +/- 0.5%. Can this be the
> reason why you get slightly different numbers?
>
>
> > we may have disagreements on the
> > parameters to use, but we agree on the numbers those parameters produce.
> >
> > (I have since modified the Google framework to include a script that
> > pulls in the sources for the needed binaries and compiles them - if you
> > want to make 100% sure people are working from the same sources, you may
> > want to rebase to a newer version of the comparision toolkit.)
> >
> > On 06/22/2013 03:41 PM, Bo Burman wrote:
> >> Hi all,
> >>
> >> We have had a look at Google's comparison between VP8 and H.264
> constrained baseline that was posted on April 3rd (
> http://www.ietf.org/mail-archive/web/rtcweb/current/msg07028.html). This
> post contains, as the one mentioned above (and if the attachments make it
> to the list), information on the exact tools and options used for encoding
> and should thus be repeatable by anyone interested.
> >>
> >> As was already stated by others on this list, one major problem is that
> Google's test involves the rate control mechanism. Typically codecs are
> measured with rate control turned off, since it acts as a huge noise on the
> measurement. Instead we propose to compare the codecs using fixed
> qp-levels. The qp-level is the quantization parameter that affects the
> rate/distortion tradeoff. Comparing using fixed qp-levels is what has been
> used when benchmarking HEVC against H.264 in the JCT-VC standardization,
> for instance. We are going to select a codec (essentially bit stream
> format), not a rate control mechanism: Once the codec is selected you can
> choose whatever rate control mechanism you wish.
> >>
> >> We used Google's excellent framework as the baseline and changed the
> parameter settings in order to make it possible to measure using fixed qp.
> We used the same sequences, but limited them to the first 10 seconds since
> they varied from 10 seconds to minutes; this also eased computation time.
> >>
> >> We used two H.264 encoder implementations: X264, which is an
> open-source codec that can operate in everything from real-time to slow,
> and JM which is the reference implementation that was used to develop
> H.264. JM is very slow but attempts to be very efficient in terms of bits
> per quality. The results were as follows:
> >>
> >> X264 baseline vs VP8: H.264 wins with 1%
> >> JM baseline vs VP8: H.264 wins with 4%
> >>
> >> Running times:
> >> X264: 1 hour 3 minutes
> >> VP8: 2 hours 0 minutes
> >> JM: order of magnitude slower
> >>
> >> It is interesting to note that the measurements are more stable in the
> new test; the variance of the percentages for the sequences is now around
> 70, down from around 700 in Google's test of April 3rd.  We believe this is
> due to the removal of the rate controller, which acts like noise on the
> measurements.
> >>
> >> We also tried setting H.264 to constrained high (no interlace and no
> B-pictures, compared to high). The results were then:
> >>
> >> X264 constrained high vs VP8: H.264 wins with 25%
> >> JM constrained high vs VP8: H.264 wins with 24%
> >>
> >> We also note that the script that Google provided to calculate the rate
> differences ("BD-rate") does not give exactly the same numbers as the
> JCT-VC-way of calculating BD-rate. The main difference is that the JM score
> for constrained high is better (around 29%) if the JCT-VC way of
> calculating BD-rate is used.
> >>
> >> In summary we think that proper testing can conclude that there is no
> clear performance advantage to any codec between VP8 and H.264 baseline.
> When comparing VP8 against H.264 constrained high on the other hand, it
> seems like there is an advantage for H.264 constrained high.
> >>
> >> The attached file includes the files necessary to reproduce the test.
> >>
> >> Best Regards,
> >>
> >> Bo Burman
> >>
> >>
> >> _______________________________________________
> >> rtcweb mailing list
> >> rtcweb@ietf.org
> >> https://www.ietf.org/mailman/listinfo/rtcweb
> >
>
> _______________________________________________
> rtcweb mailing list
> rtcweb@ietf.org
> https://www.ietf.org/mailman/listinfo/rtcweb
>