Re: [rtcweb] More H.264 vs VP8 tests

Leon Geyser <lgeyser@gmail.com> Sat, 22 June 2013 16:54 UTC

Return-Path: <lgeyser@gmail.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B597311E810C for <rtcweb@ietfa.amsl.com>; Sat, 22 Jun 2013 09:54:27 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cX8xef6dyn5N for <rtcweb@ietfa.amsl.com>; Sat, 22 Jun 2013 09:54:26 -0700 (PDT)
Received: from mail-la0-x236.google.com (mail-la0-x236.google.com [IPv6:2a00:1450:4010:c03::236]) by ietfa.amsl.com (Postfix) with ESMTP id 806A611E810B for <rtcweb@ietf.org>; Sat, 22 Jun 2013 09:54:22 -0700 (PDT)
Received: by mail-la0-f54.google.com with SMTP id ec20so8505709lab.41 for <rtcweb@ietf.org>; Sat, 22 Jun 2013 09:54:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=AQwII1HxNERAkb93TryJPpUiFiwJ3hFwCOzmVyXdnVg=; b=XC+6OAFGmjITRei0f7UbdggvgFg9LNxMnDnDjeSJFDrI6/H6iXsYPo6M0FjZJWJMpI HyuF/6Q2Ha3L6uqh8o+rF41gLPSiTu5CD3YiNhV0agCO0vE5natr5tHnaI76QX0s/+N6 Uz8+s78BGbEJZEpx67h+o3HryjCvB/X4Uz3QFfDk4sPGJMm7/6kB4F8rulEB+fsKsOgX l5TUKPmiNMoSgneePBDvL83wBXEEi/4KU2KL81xUL22bJ9iH9f5DpK3OWgyVTonwsphf lgQU9WBTQTeI+/IJMIQVVPgzuON8/lKpz9+Y8lO00vI6vWxQOa1kcXXoXmsuY59nCja0 soDw==
MIME-Version: 1.0
X-Received: by 10.152.20.136 with SMTP id n8mr8103208lae.58.1371920061098; Sat, 22 Jun 2013 09:54:21 -0700 (PDT)
Received: by 10.114.174.201 with HTTP; Sat, 22 Jun 2013 09:54:21 -0700 (PDT)
In-Reply-To: <BBE9739C2C302046BD34B42713A1E2A22DECC12F@ESESSMB105.ericsson.se>
References: <BBE9739C2C302046BD34B42713A1E2A22DECC12F@ESESSMB105.ericsson.se>
Date: Sat, 22 Jun 2013 18:54:21 +0200
Message-ID: <CAGgHUiRmAu_8R53qZnmTFHE3016sFSD+FtZ=zTG1wjBjRxL3MA@mail.gmail.com>
From: Leon Geyser <lgeyser@gmail.com>
To: Bo Burman <bo.burman@ericsson.com>
Content-Type: multipart/alternative; boundary=089e01493e2ee6ed9904dfc107df
Cc: "rtcweb@ietf.org" <rtcweb@ietf.org>
Subject: Re: [rtcweb] More H.264 vs VP8 tests
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 22 Jun 2013 16:54:27 -0000

Hi Bo,

Thanks for the additional testing done.

I am not 100% why we have to test with fixed qp-levels when WebRTC is going
to be used over the internet. I would assume that WebRTC would be used over
the internet...
My understanding of rate-control might be completely wrong. Why test modes
that might not even be used in real-world scenarios?

On 22 June 2013 15:41, Bo Burman <bo.burman@ericsson.com> wrote:

> Hi all,
>
> We have had a look at Google's comparison between VP8 and H.264
> constrained baseline that was posted on April 3rd (
> http://www.ietf.org/mail-archive/web/rtcweb/current/msg07028.html). This
> post contains, as the one mentioned above (and if the attachments make it
> to the list), information on the exact tools and options used for encoding
> and should thus be repeatable by anyone interested.
>
> As was already stated by others on this list, one major problem is that
> Google's test involves the rate control mechanism. Typically codecs are
> measured with rate control turned off, since it acts as a huge noise on the
> measurement. Instead we propose to compare the codecs using fixed
> qp-levels. The qp-level is the quantization parameter that affects the
> rate/distortion tradeoff. Comparing using fixed qp-levels is what has been
> used when benchmarking HEVC against H.264 in the JCT-VC standardization,
> for instance. We are going to select a codec (essentially bit stream
> format), not a rate control mechanism: Once the codec is selected you can
> choose whatever rate control mechanism you wish.
>
> We used Google's excellent framework as the baseline and changed the
> parameter settings in order to make it possible to measure using fixed qp.
> We used the same sequences, but limited them to the first 10 seconds since
> they varied from 10 seconds to minutes; this also eased computation time.
>
> We used two H.264 encoder implementations: X264, which is an open-source
> codec that can operate in everything from real-time to slow, and JM which
> is the reference implementation that was used to develop H.264. JM is very
> slow but attempts to be very efficient in terms of bits per quality. The
> results were as follows:
>
> X264 baseline vs VP8: H.264 wins with 1%
> JM baseline vs VP8: H.264 wins with 4%
>
> Running times:
> X264: 1 hour 3 minutes
> VP8: 2 hours 0 minutes
> JM: order of magnitude slower
>
> It is interesting to note that the measurements are more stable in the new
> test; the variance of the percentages for the sequences is now around 70,
> down from around 700 in Google's test of April 3rd.  We believe this is due
> to the removal of the rate controller, which acts like noise on the
> measurements.
>
> We also tried setting H.264 to constrained high (no interlace and no
> B-pictures, compared to high). The results were then:
>
> X264 constrained high vs VP8: H.264 wins with 25%
> JM constrained high vs VP8: H.264 wins with 24%
>
> We also note that the script that Google provided to calculate the rate
> differences ("BD-rate") does not give exactly the same numbers as the
> JCT-VC-way of calculating BD-rate. The main difference is that the JM score
> for constrained high is better (around 29%) if the JCT-VC way of
> calculating BD-rate is used.
>
> In summary we think that proper testing can conclude that there is no
> clear performance advantage to any codec between VP8 and H.264 baseline.
> When comparing VP8 against H.264 constrained high on the other hand, it
> seems like there is an advantage for H.264 constrained high.
>
> The attached file includes the files necessary to reproduce the test.
>
> Best Regards,
>
> Bo Burman
>
>
> _______________________________________________
> rtcweb mailing list
> rtcweb@ietf.org
> https://www.ietf.org/mailman/listinfo/rtcweb
>
>