Re: [rtcweb] More H.264 vs VP8 tests

John Koleszar <jkoleszar@google.com> Tue, 25 June 2013 15:36 UTC

Return-Path: <jkoleszar@google.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7914821E8091 for <rtcweb@ietfa.amsl.com>; Tue, 25 Jun 2013 08:36:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.978
X-Spam-Level:
X-Spam-Status: No, score=-1.978 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qhTwe8m4UmZx for <rtcweb@ietfa.amsl.com>; Tue, 25 Jun 2013 08:36:21 -0700 (PDT)
Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) by ietfa.amsl.com (Postfix) with ESMTP id 8B02511E80F2 for <rtcweb@ietf.org>; Tue, 25 Jun 2013 08:36:20 -0700 (PDT)
Received: by mail-wi0-f182.google.com with SMTP id m6so894706wiv.15 for <rtcweb@ietf.org>; Tue, 25 Jun 2013 08:36:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=uBTwR+bnwE/irK59ws0bVwd6cnD2skW5s9eVm3hOPCU=; b=piWEswf8OplNY0kqcibJbn3xjjaPdlnTpTYK4iReemKz/FdjEpOMTIVBb8Bzahcb5T /Ge3xvAJTA8XFfWD97upMegt1PcpKbsQ2JbCpi/aFJ140bOdilLwd5YYlITv7ebWhJ/t +HYuVyBm5aQdDn+wY5wau+XLBXDD+ltRuYSvssdFUbpbn5ojFHOiMVwwZAv2nqBD8H7M hlO0TV9iVOdOJXj/JSe1Qn6stvEuRnpmkXI0+qNovCXx2Fk1zK4ZZ70NwxTeXSu2Rvt6 W4dgRa5uJtZA0UCqSmbPMwpNiIj/Cpb8IFf++W77cLV5XJtmX0EOxbVky9r9bO6I54UC DhBA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding:x-gm-message-state; bh=uBTwR+bnwE/irK59ws0bVwd6cnD2skW5s9eVm3hOPCU=; b=Aex74bV/wRhk4VadCDpe5oWD6KBUn8ppUxOE5i0aJiWiVWcJ3I4L8o7brOmO3miXE2 Es932Wy6cEpVHCELWMrkz0kZvuk5uonbOKjNbYCcjrwrvdHq89qlgeGpqoMqiyig8EYy zY450LWrpu2YPyW58SWk3kW9O+aOGwDCJnLjJk/n4hqcaZrbILF/5PXVgxyEXQ28RNq/ 9Yvo8x50zYnAolG1VJ0LHi9e68bYQT/A4HoZkxLJ/10kMUzwII8O2UcqROuCh0LzGCQJ dUiKrO2J6xwQnHJ1Wnl7MX6dCXOr842SVIAaVzOnFaTjwMFCGDTr61QwP0rD1KPXzJJx KgVQ==
MIME-Version: 1.0
X-Received: by 10.180.77.74 with SMTP id q10mr9504235wiw.28.1372174578409; Tue, 25 Jun 2013 08:36:18 -0700 (PDT)
Received: by 10.194.138.148 with HTTP; Tue, 25 Jun 2013 08:36:18 -0700 (PDT)
In-Reply-To: <BBE9739C2C302046BD34B42713A1E2A22DECC12F@ESESSMB105.ericsson.se>
References: <BBE9739C2C302046BD34B42713A1E2A22DECC12F@ESESSMB105.ericsson.se>
Date: Tue, 25 Jun 2013 08:36:18 -0700
Message-ID: <CAAvHm8P5w_sy_cRScudpg7RytxZJRhxykmHtw1dFMB42BYF5-g@mail.gmail.com>
From: John Koleszar <jkoleszar@google.com>
To: Bo Burman <bo.burman@ericsson.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Gm-Message-State: ALoCoQmnDfR8HIPhImjJkDnIKEBPWTy1egEZ2/uTsLRo8p9kp9q+muRC3m4V2c1ijAd7FXd/3UW2lBW5EOCFCWcHDtfm6WwqdYD3r7Z1dJTYHQki2cvwHGbyIHgNDRtHgryZXu20xj073mjZiv3rKhylKuVWrtvfd6epXo+kqdtTakC6UoRXuPe4tSUGZnrV63+KYe78WENU
Cc: "rtcweb@ietf.org" <rtcweb@ietf.org>
Subject: Re: [rtcweb] More H.264 vs VP8 tests
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 25 Jun 2013 15:36:21 -0000

On Sat, Jun 22, 2013 at 6:41 AM, Bo Burman <bo.burman@ericsson.com> wrote:
> Hi all,
>
> We have had a look at Google's comparison between VP8 and H.264 constrained baseline that was posted on April 3rd (http://www.ietf.org/mail-archive/web/rtcweb/current/msg07028.html). This post contains, as the one mentioned above (and if the attachments make it to the list), information on the exact tools and options used for encoding and should thus be repeatable by anyone interested.
>
> As was already stated by others on this list, one major problem is that Google's test involves the rate control mechanism. Typically codecs are measured with rate control turned off, since it acts as a huge noise on the measurement. Instead we propose to compare the codecs using fixed qp-levels. The qp-level is the quantization parameter that affects the rate/distortion tradeoff. Comparing using fixed qp-levels is what has been used when benchmarking HEVC against H.264 in the JCT-VC standardization, for instance. We are going to select a codec (essentially bit stream format), not a rate control mechanism: Once the codec is selected you can choose whatever rate control mechanism you wish.
>
> We used Google's excellent framework as the baseline and changed the parameter settings in order to make it possible to measure using fixed qp. We used the same sequences, but limited them to the first 10 seconds since they varied from 10 seconds to minutes; this also eased computation time.
>
> We used two H.264 encoder implementations: X264, which is an open-source codec that can operate in everything from real-time to slow, and JM which is the reference implementation that was used to develop H.264. JM is very slow but attempts to be very efficient in terms of bits per quality. The results were as follows:
>
> X264 baseline vs VP8: H.264 wins with 1%
> JM baseline vs VP8: H.264 wins with 4%
>
> Running times:
> X264: 1 hour 3 minutes
> VP8: 2 hours 0 minutes
> JM: order of magnitude slower
>
> It is interesting to note that the measurements are more stable in the new test; the variance of the percentages for the sequences is now around 70, down from around 700 in Google's test of April 3rd.  We believe this is due to the removal of the rate controller, which acts like noise on the measurements.
>
> We also tried setting H.264 to constrained high (no interlace and no B-pictures, compared to high). The results were then:
>
> X264 constrained high vs VP8: H.264 wins with 25%
> JM constrained high vs VP8: H.264 wins with 24%
>
[...]

At one point, running the libvpx implementation of VP8 in fixed qp
mode would effectively limit the encoder to only a single reference
frame. I don't recall if it's still the case. Another unexpected
behavior for people trying this mode is that making libvpx bump up
against the lower quantizer without setting an approximately correct
target bitrate can also put the encoder into its "over quant" mode,
where it'll throw away additional residual trying to hit the target
bitrate. It doesn't have anything like the x264 ipratio (though I see
you set this to 1, but this is something that bites people trying
fixed-qp). This mode isn't really supported by libvpx, so I'd be
careful about taking these results as representative of the very best
VP8 can do.