Re: [rtcweb] More H.264 vs VP8 tests
Saverio Mascolo <saverio.mascolo@gmail.com> Mon, 01 July 2013 17:10 UTC
Return-Path: <saverio.mascolo@gmail.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 643C611E8208 for <rtcweb@ietfa.amsl.com>; Mon, 1 Jul 2013 10:10:09 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.299
X-Spam-Level:
X-Spam-Status: No, score=-2.299 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_8BIT_HEADER=0.3, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NmXQGIQC5uiD for <rtcweb@ietfa.amsl.com>; Mon, 1 Jul 2013 10:10:08 -0700 (PDT)
Received: from mail-ie0-x233.google.com (mail-ie0-x233.google.com [IPv6:2607:f8b0:4001:c03::233]) by ietfa.amsl.com (Postfix) with ESMTP id 5F52311E8232 for <rtcweb@ietf.org>; Mon, 1 Jul 2013 10:10:08 -0700 (PDT)
Received: by mail-ie0-f179.google.com with SMTP id c10so9833655ieb.38 for <rtcweb@ietf.org>; Mon, 01 Jul 2013 10:10:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=hKw+1HgjZmytbOMI1NASpRLo0rjsZ21PBYfqvPMn9KU=; b=Pozt72HWPJ5I0KPT3QeZcQc5zviow5FAtx4OFAGp1f9/CRyQnv4kkuMiJgwYELdoof EmZD6/Glvsoy038xBOeNtEONH414iV5p6Gp0q+5Gif1BrE2mzkKdc1rubRJULcUV+iy5 6PKRNEJtsninCXUyhvIvUDYQcv21m7i2D+Eju+bU6B9a2a1KxRX2NGFo9LR35UpsA7C/ VRCH2GX3nL1R1HnDC8WEDyAVbmCtUv6zav4aA/frRQiclOOcmVovjELoJ83tBR40W7qM 0VFPeGzQ1Ovg3zGlxuLuRCK7d8UTRgQA3xxvPw53Z8uLcKBaQjvSxGl5SsnqywGnguG/ IEPA==
X-Received: by 10.50.122.100 with SMTP id lr4mr16563829igb.18.1372698607854; Mon, 01 Jul 2013 10:10:07 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.64.138.166 with HTTP; Mon, 1 Jul 2013 10:09:47 -0700 (PDT)
In-Reply-To: <1447FA0C20ED5147A1AA0EF02890A64B1C308D3B@ESESSMB209.ericsson.se>
References: <BBE9739C2C302046BD34B42713A1E2A22DECC12F@ESESSMB105.ericsson.se> <51C96E36.2000907@alvestrand.no> <1447FA0C20ED5147A1AA0EF02890A64B1C308D3B@ESESSMB209.ericsson.se>
From: Saverio Mascolo <saverio.mascolo@gmail.com>
Date: Mon, 01 Jul 2013 19:09:47 +0200
Message-ID: <CAK1jYfdDRCJjGNwU1nimmKEqThSQSn9=7zEJ3PJXdru+MAT1fQ@mail.gmail.com>
To: Stefan Håkansson LK <stefan.lk.hakansson@ericsson.com>
Content-Type: multipart/alternative; boundary="001a11c1f7a4e7aa6a04e0764cad"
Cc: "rtcweb@ietf.org" <rtcweb@ietf.org>
Subject: Re: [rtcweb] More H.264 vs VP8 tests
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 01 Jul 2013 17:10:09 -0000
an evaluation of how h264 and vp8 are able to match a changing quality would be interesting On Sat, Jun 29, 2013 at 3:17 PM, Stefan Håkansson LK < stefan.lk.hakansson@ericsson.com> wrote: > On 6/25/13 12:17 PM, Harald Alvestrand wrote: > > Again - thanks for releasing this openly! > > > > I ran the scripts (with a few tweaks; you run on a system where sh is > > bash, not dash, for instance), and got the same numbers within +/- 0.5% > > (probably some binary version skew); > > I re-run the scripts on another computer with another OS today, and I > get exactly the same results as Bo sent out. I noted however that if the > input clips are not cut at 10s (but used in their entire length) the > results get slightly different, but within +/- 0.5%. Can this be the > reason why you get slightly different numbers? > > > > we may have disagreements on the > > parameters to use, but we agree on the numbers those parameters produce. > > > > (I have since modified the Google framework to include a script that > > pulls in the sources for the needed binaries and compiles them - if you > > want to make 100% sure people are working from the same sources, you may > > want to rebase to a newer version of the comparision toolkit.) > > > > On 06/22/2013 03:41 PM, Bo Burman wrote: > >> Hi all, > >> > >> We have had a look at Google's comparison between VP8 and H.264 > constrained baseline that was posted on April 3rd ( > http://www.ietf.org/mail-archive/web/rtcweb/current/msg07028.html) This > post contains, as the one mentioned above (and if the attachments make it > to the list), information on the exact tools and options used for encoding > and should thus be repeatable by anyone interested. > >> > >> As was already stated by others on this list, one major problem is that > Google's test involves the rate control mechanism. Typically codecs are > measured with rate control turned off, since it acts as a huge noise on the > measurement. Instead we propose to compare the codecs using fixed > qp-levels. The qp-level is the quantization parameter that affects the > rate/distortion tradeoff. Comparing using fixed qp-levels is what has been > used when benchmarking HEVC against H.264 in the JCT-VC standardization, > for instance. We are going to select a codec (essentially bit stream > format), not a rate control mechanism: Once the codec is selected you can > choose whatever rate control mechanism you wish. > >> > >> We used Google's excellent framework as the baseline and changed the > parameter settings in order to make it possible to measure using fixed qp. > We used the same sequences, but limited them to the first 10 seconds since > they varied from 10 seconds to minutes; this also eased computation time. > >> > >> We used two H.264 encoder implementations: X264, which is an > open-source codec that can operate in everything from real-time to slow, > and JM which is the reference implementation that was used to develop > H.264. JM is very slow but attempts to be very efficient in terms of bits > per quality. The results were as follows: > >> > >> X264 baseline vs VP8: H.264 wins with 1% > >> JM baseline vs VP8: H.264 wins with 4% > >> > >> Running times: > >> X264: 1 hour 3 minutes > >> VP8: 2 hours 0 minutes > >> JM: order of magnitude slower > >> > >> It is interesting to note that the measurements are more stable in the > new test; the variance of the percentages for the sequences is now around > 70, down from around 700 in Google's test of April 3rd. We believe this is > due to the removal of the rate controller, which acts like noise on the > measurements. > >> > >> We also tried setting H.264 to constrained high (no interlace and no > B-pictures, compared to high). The results were then: > >> > >> X264 constrained high vs VP8: H.264 wins with 25% > >> JM constrained high vs VP8: H.264 wins with 24% > >> > >> We also note that the script that Google provided to calculate the rate > differences ("BD-rate") does not give exactly the same numbers as the > JCT-VC-way of calculating BD-rate. The main difference is that the JM score > for constrained high is better (around 29%) if the JCT-VC way of > calculating BD-rate is used. > >> > >> In summary we think that proper testing can conclude that there is no > clear performance advantage to any codec between VP8 and H.264 baseline. > When comparing VP8 against H.264 constrained high on the other hand, it > seems like there is an advantage for H.264 constrained high. > >> > >> The attached file includes the files necessary to reproduce the test. > >> > >> Best Regards, > >> > >> Bo Burman > >> > >> > >> _______________________________________________ > >> rtcweb mailing list > >> rtcweb@ietf.org > >> https://www.ietf.org/mailman/listinfo/rtcweb > > > > _______________________________________________ > rtcweb mailing list > rtcweb@ietf.org > https://www.ietf.org/mailman/listinfo/rtcweb >
- [rtcweb] More H.264 vs VP8 tests Bo Burman
- Re: [rtcweb] More H.264 vs VP8 tests Leon Geyser
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Harald Alvestrand
- Re: [rtcweb] More H.264 vs VP8 tests Harald Alvestrand
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests John Koleszar
- Re: [rtcweb] More H.264 vs VP8 tests Eggert, Lars
- Re: [rtcweb] More H.264 vs VP8 tests Mo Zanaty (mzanaty)
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Saverio Mascolo
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Mo Zanaty (mzanaty)