Re: [rtcweb] More H.264 vs VP8 tests
Leon Geyser <lgeyser@gmail.com> Sat, 22 June 2013 16:54 UTC
Return-Path: <lgeyser@gmail.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B597311E810C for <rtcweb@ietfa.amsl.com>; Sat, 22 Jun 2013 09:54:27 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cX8xef6dyn5N for <rtcweb@ietfa.amsl.com>; Sat, 22 Jun 2013 09:54:26 -0700 (PDT)
Received: from mail-la0-x236.google.com (mail-la0-x236.google.com [IPv6:2a00:1450:4010:c03::236]) by ietfa.amsl.com (Postfix) with ESMTP id 806A611E810B for <rtcweb@ietf.org>; Sat, 22 Jun 2013 09:54:22 -0700 (PDT)
Received: by mail-la0-f54.google.com with SMTP id ec20so8505709lab.41 for <rtcweb@ietf.org>; Sat, 22 Jun 2013 09:54:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=AQwII1HxNERAkb93TryJPpUiFiwJ3hFwCOzmVyXdnVg=; b=XC+6OAFGmjITRei0f7UbdggvgFg9LNxMnDnDjeSJFDrI6/H6iXsYPo6M0FjZJWJMpI HyuF/6Q2Ha3L6uqh8o+rF41gLPSiTu5CD3YiNhV0agCO0vE5natr5tHnaI76QX0s/+N6 Uz8+s78BGbEJZEpx67h+o3HryjCvB/X4Uz3QFfDk4sPGJMm7/6kB4F8rulEB+fsKsOgX l5TUKPmiNMoSgneePBDvL83wBXEEi/4KU2KL81xUL22bJ9iH9f5DpK3OWgyVTonwsphf lgQU9WBTQTeI+/IJMIQVVPgzuON8/lKpz9+Y8lO00vI6vWxQOa1kcXXoXmsuY59nCja0 soDw==
MIME-Version: 1.0
X-Received: by 10.152.20.136 with SMTP id n8mr8103208lae.58.1371920061098; Sat, 22 Jun 2013 09:54:21 -0700 (PDT)
Received: by 10.114.174.201 with HTTP; Sat, 22 Jun 2013 09:54:21 -0700 (PDT)
In-Reply-To: <BBE9739C2C302046BD34B42713A1E2A22DECC12F@ESESSMB105.ericsson.se>
References: <BBE9739C2C302046BD34B42713A1E2A22DECC12F@ESESSMB105.ericsson.se>
Date: Sat, 22 Jun 2013 18:54:21 +0200
Message-ID: <CAGgHUiRmAu_8R53qZnmTFHE3016sFSD+FtZ=zTG1wjBjRxL3MA@mail.gmail.com>
From: Leon Geyser <lgeyser@gmail.com>
To: Bo Burman <bo.burman@ericsson.com>
Content-Type: multipart/alternative; boundary="089e01493e2ee6ed9904dfc107df"
Cc: "rtcweb@ietf.org" <rtcweb@ietf.org>
Subject: Re: [rtcweb] More H.264 vs VP8 tests
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 22 Jun 2013 16:54:27 -0000
Hi Bo, Thanks for the additional testing done. I am not 100% why we have to test with fixed qp-levels when WebRTC is going to be used over the internet. I would assume that WebRTC would be used over the internet... My understanding of rate-control might be completely wrong. Why test modes that might not even be used in real-world scenarios? On 22 June 2013 15:41, Bo Burman <bo.burman@ericsson.com> wrote: > Hi all, > > We have had a look at Google's comparison between VP8 and H.264 > constrained baseline that was posted on April 3rd ( > http://www.ietf.org/mail-archive/web/rtcweb/current/msg07028.html). This > post contains, as the one mentioned above (and if the attachments make it > to the list), information on the exact tools and options used for encoding > and should thus be repeatable by anyone interested. > > As was already stated by others on this list, one major problem is that > Google's test involves the rate control mechanism. Typically codecs are > measured with rate control turned off, since it acts as a huge noise on the > measurement. Instead we propose to compare the codecs using fixed > qp-levels. The qp-level is the quantization parameter that affects the > rate/distortion tradeoff. Comparing using fixed qp-levels is what has been > used when benchmarking HEVC against H.264 in the JCT-VC standardization, > for instance. We are going to select a codec (essentially bit stream > format), not a rate control mechanism: Once the codec is selected you can > choose whatever rate control mechanism you wish. > > We used Google's excellent framework as the baseline and changed the > parameter settings in order to make it possible to measure using fixed qp. > We used the same sequences, but limited them to the first 10 seconds since > they varied from 10 seconds to minutes; this also eased computation time. > > We used two H.264 encoder implementations: X264, which is an open-source > codec that can operate in everything from real-time to slow, and JM which > is the reference implementation that was used to develop H.264. JM is very > slow but attempts to be very efficient in terms of bits per quality. The > results were as follows: > > X264 baseline vs VP8: H.264 wins with 1% > JM baseline vs VP8: H.264 wins with 4% > > Running times: > X264: 1 hour 3 minutes > VP8: 2 hours 0 minutes > JM: order of magnitude slower > > It is interesting to note that the measurements are more stable in the new > test; the variance of the percentages for the sequences is now around 70, > down from around 700 in Google's test of April 3rd. We believe this is due > to the removal of the rate controller, which acts like noise on the > measurements. > > We also tried setting H.264 to constrained high (no interlace and no > B-pictures, compared to high). The results were then: > > X264 constrained high vs VP8: H.264 wins with 25% > JM constrained high vs VP8: H.264 wins with 24% > > We also note that the script that Google provided to calculate the rate > differences ("BD-rate") does not give exactly the same numbers as the > JCT-VC-way of calculating BD-rate. The main difference is that the JM score > for constrained high is better (around 29%) if the JCT-VC way of > calculating BD-rate is used. > > In summary we think that proper testing can conclude that there is no > clear performance advantage to any codec between VP8 and H.264 baseline. > When comparing VP8 against H.264 constrained high on the other hand, it > seems like there is an advantage for H.264 constrained high. > > The attached file includes the files necessary to reproduce the test. > > Best Regards, > > Bo Burman > > > _______________________________________________ > rtcweb mailing list > rtcweb@ietf.org > https://www.ietf.org/mailman/listinfo/rtcweb > >
- [rtcweb] More H.264 vs VP8 tests Bo Burman
- Re: [rtcweb] More H.264 vs VP8 tests Leon Geyser
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Harald Alvestrand
- Re: [rtcweb] More H.264 vs VP8 tests Harald Alvestrand
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests John Koleszar
- Re: [rtcweb] More H.264 vs VP8 tests Eggert, Lars
- Re: [rtcweb] More H.264 vs VP8 tests Mo Zanaty (mzanaty)
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Saverio Mascolo
- Re: [rtcweb] More H.264 vs VP8 tests Stefan Håkansson LK
- Re: [rtcweb] More H.264 vs VP8 tests Mo Zanaty (mzanaty)