Re: [rtcweb] More H.264 vs VP8 tests

Leon Geyser <> Sat, 22 June 2013 16:54 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id B597311E810C for <>; Sat, 22 Jun 2013 09:54:27 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, NO_RELAYS=-0.001]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id cX8xef6dyn5N for <>; Sat, 22 Jun 2013 09:54:26 -0700 (PDT)
Received: from ( [IPv6:2a00:1450:4010:c03::236]) by (Postfix) with ESMTP id 806A611E810B for <>; Sat, 22 Jun 2013 09:54:22 -0700 (PDT)
Received: by with SMTP id ec20so8505709lab.41 for <>; Sat, 22 Jun 2013 09:54:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=AQwII1HxNERAkb93TryJPpUiFiwJ3hFwCOzmVyXdnVg=; b=XC+6OAFGmjITRei0f7UbdggvgFg9LNxMnDnDjeSJFDrI6/H6iXsYPo6M0FjZJWJMpI HyuF/6Q2Ha3L6uqh8o+rF41gLPSiTu5CD3YiNhV0agCO0vE5natr5tHnaI76QX0s/+N6 Uz8+s78BGbEJZEpx67h+o3HryjCvB/X4Uz3QFfDk4sPGJMm7/6kB4F8rulEB+fsKsOgX l5TUKPmiNMoSgneePBDvL83wBXEEi/4KU2KL81xUL22bJ9iH9f5DpK3OWgyVTonwsphf lgQU9WBTQTeI+/IJMIQVVPgzuON8/lKpz9+Y8lO00vI6vWxQOa1kcXXoXmsuY59nCja0 soDw==
MIME-Version: 1.0
X-Received: by with SMTP id n8mr8103208lae.58.1371920061098; Sat, 22 Jun 2013 09:54:21 -0700 (PDT)
Received: by with HTTP; Sat, 22 Jun 2013 09:54:21 -0700 (PDT)
In-Reply-To: <>
References: <>
Date: Sat, 22 Jun 2013 18:54:21 +0200
Message-ID: <>
From: Leon Geyser <>
To: Bo Burman <>
Content-Type: multipart/alternative; boundary="089e01493e2ee6ed9904dfc107df"
Cc: "" <>
Subject: Re: [rtcweb] More H.264 vs VP8 tests
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Sat, 22 Jun 2013 16:54:27 -0000

Hi Bo,

Thanks for the additional testing done.

I am not 100% why we have to test with fixed qp-levels when WebRTC is going
to be used over the internet. I would assume that WebRTC would be used over
the internet...
My understanding of rate-control might be completely wrong. Why test modes
that might not even be used in real-world scenarios?

On 22 June 2013 15:41, Bo Burman <> wrote:

> Hi all,
> We have had a look at Google's comparison between VP8 and H.264
> constrained baseline that was posted on April 3rd (
> This
> post contains, as the one mentioned above (and if the attachments make it
> to the list), information on the exact tools and options used for encoding
> and should thus be repeatable by anyone interested.
> As was already stated by others on this list, one major problem is that
> Google's test involves the rate control mechanism. Typically codecs are
> measured with rate control turned off, since it acts as a huge noise on the
> measurement. Instead we propose to compare the codecs using fixed
> qp-levels. The qp-level is the quantization parameter that affects the
> rate/distortion tradeoff. Comparing using fixed qp-levels is what has been
> used when benchmarking HEVC against H.264 in the JCT-VC standardization,
> for instance. We are going to select a codec (essentially bit stream
> format), not a rate control mechanism: Once the codec is selected you can
> choose whatever rate control mechanism you wish.
> We used Google's excellent framework as the baseline and changed the
> parameter settings in order to make it possible to measure using fixed qp.
> We used the same sequences, but limited them to the first 10 seconds since
> they varied from 10 seconds to minutes; this also eased computation time.
> We used two H.264 encoder implementations: X264, which is an open-source
> codec that can operate in everything from real-time to slow, and JM which
> is the reference implementation that was used to develop H.264. JM is very
> slow but attempts to be very efficient in terms of bits per quality. The
> results were as follows:
> X264 baseline vs VP8: H.264 wins with 1%
> JM baseline vs VP8: H.264 wins with 4%
> Running times:
> X264: 1 hour 3 minutes
> VP8: 2 hours 0 minutes
> JM: order of magnitude slower
> It is interesting to note that the measurements are more stable in the new
> test; the variance of the percentages for the sequences is now around 70,
> down from around 700 in Google's test of April 3rd.  We believe this is due
> to the removal of the rate controller, which acts like noise on the
> measurements.
> We also tried setting H.264 to constrained high (no interlace and no
> B-pictures, compared to high). The results were then:
> X264 constrained high vs VP8: H.264 wins with 25%
> JM constrained high vs VP8: H.264 wins with 24%
> We also note that the script that Google provided to calculate the rate
> differences ("BD-rate") does not give exactly the same numbers as the
> JCT-VC-way of calculating BD-rate. The main difference is that the JM score
> for constrained high is better (around 29%) if the JCT-VC way of
> calculating BD-rate is used.
> In summary we think that proper testing can conclude that there is no
> clear performance advantage to any codec between VP8 and H.264 baseline.
> When comparing VP8 against H.264 constrained high on the other hand, it
> seems like there is an advantage for H.264 constrained high.
> The attached file includes the files necessary to reproduce the test.
> Best Regards,
> Bo Burman
> _______________________________________________
> rtcweb mailing list