Re: [rtcweb] New VP8 vs H.264 tests uploaded
Thomas Davies <thdavies@cisco.com> Thu, 04 April 2013 15:05 UTC
Return-Path: <thdavies@cisco.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EDD0A21F8556 for <rtcweb@ietfa.amsl.com>; Thu, 4 Apr 2013 08:05:40 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.598
X-Spam-Level:
X-Spam-Status: No, score=-10.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lpco5RTEoTwE for <rtcweb@ietfa.amsl.com>; Thu, 4 Apr 2013 08:05:36 -0700 (PDT)
Received: from ams-iport-1.cisco.com (ams-iport-1.cisco.com [144.254.224.140]) by ietfa.amsl.com (Postfix) with ESMTP id D2C0D21F84CE for <rtcweb@ietf.org>; Thu, 4 Apr 2013 08:05:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=21102; q=dns/txt; s=iport; t=1365087936; x=1366297536; h=message-id:date:from:mime-version:to:subject:references: in-reply-to; bh=rbaqjb/x61yW1IwZz4niGUZHmYanYQSv7gOsRf2iKRk=; b=P8z3wRv/7EnrHQcmcLoWyuVZGDUIZWRHj+qGDP9xhMO/m07vpJgna88Y SF6AdjfKITlTMm6l0AAI7B0lxNM2ODCKlni+3NwxTRkOvxWXZ/l6+MxV0 BZ76G9GN8qvI0bGG2g3FzRcf9mwfPpIel6w21SpWZp368lvyrAR7sVv8y Q=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AikFAKKVXVGQ/khL/2dsb2JhbABDgkJENokGuAGBABZ0gh8BAQEEAQEBaAECCAIRCxgJFg8JAwIBAgEVMAcMBgIBAYgQDMEGjyKDQAOWboEghGKLC4MMOw
X-IronPort-AV: E=Sophos; i="4.87,409,1363132800"; d="scan'208,217"; a="152492969"
Received: from ams-core-2.cisco.com ([144.254.72.75]) by ams-iport-1.cisco.com with ESMTP; 04 Apr 2013 15:05:33 +0000
Received: from [10.47.196.175] ([10.47.196.175]) by ams-core-2.cisco.com (8.14.5/8.14.5) with ESMTP id r34F5Vu3020966; Thu, 4 Apr 2013 15:05:32 GMT
Message-ID: <515D96A2.1000602@cisco.com>
Date: Thu, 04 Apr 2013 16:05:06 +0100
From: Thomas Davies <thdavies@cisco.com>
Organization: Cisco Systems
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:11.0) Gecko/20120410 Thunderbird/11.0.1
MIME-Version: 1.0
To: harald@alvestrand.no, rtcweb@ietf.org
References: <CAPVCLWbajJNS-DbXS-AJjakwovBKhhpXAmBaR_LYKjCyk7UnYg@mail.gmail.com> <515D3FA1.6050305@gmail.com>
In-Reply-To: <515D3FA1.6050305@gmail.com>
Content-Type: multipart/alternative; boundary="------------010801090409010400000800"
Subject: Re: [rtcweb] New VP8 vs H.264 tests uploaded
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 04 Apr 2013 15:05:41 -0000
Harald, I think there are quite a few problems with the comparison you have posted. 1. Looking at the sequences there is a very major difference between the initial intra frame qualities. When I encode just one frame of sequence gipsrecomotion using the parameters in the script at 1Mb/s then the intra frame is 3 times larger with vp8 than with x264. With video conferencing content, the quality of the initial I frame has a big impact that can last for many seconds - certainly the length of these clips. You can easily get gains by increasing the quality difference between an I frame and subsequent frames. x264 seems to have a policy of initially undershooting the bitrate substantially and ramping up, whereas vpxenc has a different approach. During this ramp up period the quality is very much worse. I can't find a way to persuade x264 to behave differently. This is a good illustration of why including rate control in comparisons is a bad idea. 2. Likewise, looking at the individual frame sizes, it seems vpxenc is using a quality hierarchy with a length of 8 ("hiercharchical-P") where every 8th frame is about 4x bigger than the others. x264 has a constant target per frame. Hierarchical P frames are a really good idea, and can easily get you 10-20% gain with a big separation like this, at a cost in latency. Again I don't know how to make x264 do this, but the technique is applicable to any codec and is used in the JM reference. 3. The x264 settings are a bit of a black art, but appear not to be ideal after all. I am definitely no expert but I found that when encoding gipsrecomotion at 1Mb/s: - setting --threads 1 improves quality by a full 1dB (vpxenc seems to run single threaded by default) - reducing the number of references from 3 to 2 (--ref 2) reduces the load very substantially at very little loss (0.2dB or so). So with --threads 1 --ref 2, I found x264 ran more than 2x faster than vpxenc for this data point and had much better quality than before. vpxenc is still better (about 1dB), but very possibly within the range of hierarchical P coding improvements. Incidentally, I don't think that x264 performs particularly well at these high complexity settings, at least for video conferencing, no doubt as other more practical settings have been targeted. x264 appears to have a quality ceiling that the JM does not have. 4. Another (smaller) issue is that the reported PSNR is combined luma and chroma over all frames. It's relatively easy to improve chroma PSNR at a small cost in bits, and usually it is best to ignore chroma PSNR or (possibly) give it a small weight. The arithmetic mean of frame PSNRs is generally used rather than the PSNR of the whole sequence, also. I would very much like separate component PSNRs in tests. The figures I quote above are luma PSNR. If the purpose is to show whether vp8 is superior as a *technology* to h264 CBP, then I think the comparison should use the best settings you have (ideally with a special full-on non-real time implementation) and test against the JM reference encoder. Ideally you would use the same or similar GOP structures, number of references, prediction and QP hierarchies. Comparing different real-time implementations of different codecs trying to do high quality coding with different GOP structures and using rate control with different strategies is just a waste of time. The first two elements in the list above are alone worth a very significant amount of bit rate. On the other hand, a quick perusal of the actual tools would suggest that vp8 and h264 CBP are likely "comparable" and the variation between implementations of the same technology would be bigger than the variation between the technologies. If we could agree that then a lot of time could be saved. best regards Thomas On 04/04/13 09:53, Sergio Garcia Murillo wrote: > Hi Adrian, > > Could you explain how the encoding parametrization is comparable? > > x264 --nal-hrd cbr --vbv-maxrate ${rate} --vbv-bufsize ${rate} \ > --vbv-init 0.8 --bitrate ${rate} --fps ${frame_rate} \ > --profile baseline --no-scenecut --keyint infinite --preset > veryslow \ > --input-res ${width}x${height} \ > --tune psnr \ > -o ./encoded_clips/h264/${clip_stem}_${rate}kbps.mkv ${filename} \ > 2> ./logs/h264/${clip_stem}_${rate}kbps.txt > > vs: > > ./bin/vpxenc --lag-in-frames=0 --target-bitrate=${rate} > --kf-min-dist=3000 \ > --kf-max-dist=3000 --cpu-used=0 --fps=${frame_rate}/1 > --static-thresh=0 \ > --token-parts=1 --drop-frame=0 --end-usage=cbr --min-q=2 > --max-q=56 \ > --undershoot-pct=100 --overshoot-pct=15 --buf-sz=1000 \ > --buf-initial-sz=800 --buf-optimal-sz=1000 --max-intra-rate=1200 \ > --resize-allowed=0 --drop-frame=0 --passes=1 --good > --noise-sensitivity=0 \ > -w ${width} -h ${height} ${filename} --codec=vp8 \ > -o ./encoded_clips/vp8/${clip_stem}_${rate}kbps.webm \ > &>./logs/vp8/${clip_stem}_${rate}kbps.txt > > Best regards > Sergio > > El 03/04/2013 18:20, Adrian Grange escribió: >> We have uploaded a new set of test results comparing VP8 to H.264. >> This latest set contains fixes for some of the problems in the >> previous set. We would like to extend our thanks to those who made >> suggestions as to how we could improve our methodology and encourage >> suggestions as to how we can make further improvements. >> >> In these tests we run x264 with the "veryslow" preset and VP8 with >> the "good, speed 0" setting in an attempt to produce comparable results. >> >> An overview of our results is available as follows: >> >> - A Quality comparison (psnr): >> http://downloads.webmproject.org/ietf_tests/vp8_vs_h264_quality.html >> >> - An Encode Speed comparison: >> http://downloads.webmproject.org/ietf_tests/vp8_vs_h264_speed.html >> >> - A comparison of the aggregate time required to decode all of the >> clips in the test: >> http://downloads.webmproject.org/ietf_tests/vp8vsh264-decodetime.txt >> >> All of our test scripts can either be downloaded from: >> http://downloads.webmproject.org/ietf_tests/vp8_vs_h264.tar.xz >> or checked out of our git/gerrit repository: >> git clone http://git.chromium.org/webm/vpx_codec_comparison.git >> >> The file README.txt, contained within, presents details of how to >> build and run the tests. >> >> The compressed video files--the output from the quality tests--can >> also be downloaded: >> >> VP8: >> http://downloads.webmproject.org/ietf_tests/vp8_videos >> <http://downloads.webmproject.org/ietf_tests/vp8_videos/>/index.html >> >> H.264: >> http://downloads.webmproject.org/ietf_tests/h264_videos/index.html >> >> Adrian Grange >> >> >> >> >> >> >> >> _______________________________________________ >> rtcweb mailing list >> rtcweb@ietf.org >> https://www.ietf.org/mailman/listinfo/rtcweb > > > > _______________________________________________ > rtcweb mailing list > rtcweb@ietf.org > https://www.ietf.org/mailman/listinfo/rtcweb
- [rtcweb] New VP8 vs H.264 tests uploaded Adrian Grange
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Sergio Garcia Murillo
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Harald Alvestrand
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Luca De Cicco
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Harald Alvestrand
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Luca De Cicco
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Thomas Davies
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Matthew Kaufman
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Adam Roach
- [rtcweb] Fundamental asymmetry [was Re: New VP8 v… Marc Petit-Huguenin
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Leon Geyser
- Re: [rtcweb] New VP8 vs H.264 tests uploaded (UNC… Roy, Radhika R CIV USARMY (US)
- Re: [rtcweb] New VP8 vs H.264 tests uploaded (UNC… Thomas Davies (thdavies)
- Re: [rtcweb] New VP8 vs H.264 tests uploaded (UNC… Harald Alvestrand
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Harald Alvestrand
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Mo Zanaty (mzanaty)
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Mo Zanaty (mzanaty)
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Sergio Garcia Murillo
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Harald Alvestrand
- Re: [rtcweb] New VP8 vs H.264 tests uploaded (UNC… Roy, Radhika R CIV USARMY (US)
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Kieran Kunhya
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Mo Zanaty (mzanaty)
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Kieran Kunhya
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Randell Jesup
- Re: [rtcweb] New VP8 vs H.264 tests uploaded Leon Geyser