Re: [rtcweb] New VP8 vs H.264 tests uploaded

Thomas Davies <thdavies@cisco.com> Thu, 04 April 2013 15:05 UTC

Return-Path: <thdavies@cisco.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EDD0A21F8556 for <rtcweb@ietfa.amsl.com>; Thu, 4 Apr 2013 08:05:40 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.598
X-Spam-Level:
X-Spam-Status: No, score=-10.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lpco5RTEoTwE for <rtcweb@ietfa.amsl.com>; Thu, 4 Apr 2013 08:05:36 -0700 (PDT)
Received: from ams-iport-1.cisco.com (ams-iport-1.cisco.com [144.254.224.140]) by ietfa.amsl.com (Postfix) with ESMTP id D2C0D21F84CE for <rtcweb@ietf.org>; Thu, 4 Apr 2013 08:05:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=21102; q=dns/txt; s=iport; t=1365087936; x=1366297536; h=message-id:date:from:mime-version:to:subject:references: in-reply-to; bh=rbaqjb/x61yW1IwZz4niGUZHmYanYQSv7gOsRf2iKRk=; b=P8z3wRv/7EnrHQcmcLoWyuVZGDUIZWRHj+qGDP9xhMO/m07vpJgna88Y SF6AdjfKITlTMm6l0AAI7B0lxNM2ODCKlni+3NwxTRkOvxWXZ/l6+MxV0 BZ76G9GN8qvI0bGG2g3FzRcf9mwfPpIel6w21SpWZp368lvyrAR7sVv8y Q=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AikFAKKVXVGQ/khL/2dsb2JhbABDgkJENokGuAGBABZ0gh8BAQEEAQEBaAECCAIRCxgJFg8JAwIBAgEVMAcMBgIBAYgQDMEGjyKDQAOWboEghGKLC4MMOw
X-IronPort-AV: E=Sophos; i="4.87,409,1363132800"; d="scan'208,217"; a="152492969"
Received: from ams-core-2.cisco.com ([144.254.72.75]) by ams-iport-1.cisco.com with ESMTP; 04 Apr 2013 15:05:33 +0000
Received: from [10.47.196.175] ([10.47.196.175]) by ams-core-2.cisco.com (8.14.5/8.14.5) with ESMTP id r34F5Vu3020966; Thu, 4 Apr 2013 15:05:32 GMT
Message-ID: <515D96A2.1000602@cisco.com>
Date: Thu, 04 Apr 2013 16:05:06 +0100
From: Thomas Davies <thdavies@cisco.com>
Organization: Cisco Systems
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:11.0) Gecko/20120410 Thunderbird/11.0.1
MIME-Version: 1.0
To: harald@alvestrand.no, rtcweb@ietf.org
References: <CAPVCLWbajJNS-DbXS-AJjakwovBKhhpXAmBaR_LYKjCyk7UnYg@mail.gmail.com> <515D3FA1.6050305@gmail.com>
In-Reply-To: <515D3FA1.6050305@gmail.com>
Content-Type: multipart/alternative; boundary="------------010801090409010400000800"
Subject: Re: [rtcweb] New VP8 vs H.264 tests uploaded
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 04 Apr 2013 15:05:41 -0000

Harald,

I think there are quite a few problems with the comparison you have posted.

1. Looking at the sequences there is a very major difference between the 
initial intra frame qualities. When I encode just one frame of sequence 
gipsrecomotion using the parameters in the script at 1Mb/s then the 
intra frame is 3 times larger with vp8 than with x264.

With video conferencing content, the quality of the initial I frame has 
a big impact that can last for many seconds - certainly the length of 
these clips. You can easily get gains by increasing the quality 
difference between an I frame and subsequent frames.

x264 seems to have a policy of initially undershooting the bitrate 
substantially and ramping up, whereas vpxenc has a different approach. 
During this ramp up period the quality is very much worse. I can't find 
a way to persuade x264 to behave differently.

This is a good illustration of why including rate control in comparisons 
is a bad idea.

2. Likewise, looking at the individual frame sizes, it seems vpxenc is 
using a quality hierarchy with a length of 8 ("hiercharchical-P") where 
every 8th frame is about 4x bigger than the others. x264 has a constant 
target per frame.

Hierarchical P frames are a really good idea, and can easily get you 
10-20% gain with a big separation like this, at a cost in latency. Again 
I don't know how to make x264 do this, but the technique is applicable 
to any codec and is used in the JM reference.

3. The x264 settings are a bit of a black art, but appear not to be 
ideal after all. I am definitely no expert but I found that when 
encoding gipsrecomotion at 1Mb/s:

- setting --threads 1 improves quality by a full 1dB (vpxenc seems to 
run single threaded by default)
- reducing the number of references from 3 to 2 (--ref 2) reduces the 
load very substantially at very little loss (0.2dB or so).

So with --threads 1 --ref 2, I found x264 ran more than 2x faster than 
vpxenc for this data point and had much better quality than before. 
vpxenc is still better (about 1dB), but very possibly within the range 
of hierarchical P coding improvements.

Incidentally, I don't think that x264 performs particularly well at 
these high complexity settings, at least for video conferencing, no 
doubt as other more practical settings have been targeted. x264 appears 
to have a quality ceiling that the JM does not have.

4. Another (smaller) issue is that the reported PSNR is combined luma 
and chroma over all frames. It's relatively easy to improve chroma PSNR 
at a small cost in bits, and usually it is best to ignore chroma PSNR or 
(possibly) give it a small weight. The arithmetic mean of frame PSNRs is 
generally used rather than the PSNR of the whole sequence, also. I would 
very much like separate component PSNRs in tests. The figures I quote 
above are luma PSNR.

If the purpose is to show whether vp8 is superior as a *technology* to 
h264 CBP, then I think the comparison should use the best settings you 
have (ideally with a special full-on non-real time implementation) and 
test against the JM reference encoder. Ideally you would use the same or 
similar GOP structures, number of references, prediction and QP 
hierarchies.

Comparing different real-time implementations of different codecs trying 
to do high quality coding with different GOP structures and using rate 
control with different strategies is just a waste of time. The first two 
elements in the list above are alone worth a very significant amount of 
bit rate.

On the other hand, a quick perusal of the actual tools would suggest 
that vp8 and h264 CBP are likely "comparable" and the variation between 
implementations of the same technology would be bigger than the 
variation between the technologies. If we could agree that then a lot of 
time could be saved.

best regards

Thomas




On 04/04/13 09:53, Sergio Garcia Murillo wrote:
> Hi Adrian,
>
> Could you explain how the encoding parametrization is comparable?
>
> x264 --nal-hrd cbr --vbv-maxrate ${rate} --vbv-bufsize ${rate} \
>       --vbv-init 0.8 --bitrate ${rate} --fps ${frame_rate} \
>       --profile baseline --no-scenecut --keyint infinite --preset 
> veryslow \
>       --input-res ${width}x${height} \
>       --tune psnr \
>       -o ./encoded_clips/h264/${clip_stem}_${rate}kbps.mkv ${filename} \
>       2> ./logs/h264/${clip_stem}_${rate}kbps.txt
>
> vs:
>
>  ./bin/vpxenc --lag-in-frames=0 --target-bitrate=${rate} 
> --kf-min-dist=3000 \
>       --kf-max-dist=3000 --cpu-used=0 --fps=${frame_rate}/1 
> --static-thresh=0 \
>       --token-parts=1 --drop-frame=0 --end-usage=cbr --min-q=2 
> --max-q=56 \
>       --undershoot-pct=100 --overshoot-pct=15 --buf-sz=1000 \
>       --buf-initial-sz=800 --buf-optimal-sz=1000 --max-intra-rate=1200 \
>       --resize-allowed=0 --drop-frame=0 --passes=1 --good 
> --noise-sensitivity=0 \
>       -w ${width} -h ${height} ${filename} --codec=vp8 \
>       -o ./encoded_clips/vp8/${clip_stem}_${rate}kbps.webm \
> &>./logs/vp8/${clip_stem}_${rate}kbps.txt
>
> Best regards
> Sergio
>
> El 03/04/2013 18:20, Adrian Grange escribió:
>> We have uploaded a new set of test results comparing VP8 to H.264. 
>> This latest set contains fixes for some of the problems in the 
>> previous set. We would like to extend our thanks to those who made 
>> suggestions as to how we could improve our methodology and encourage 
>> suggestions as to how we can make further improvements.
>>
>> In these tests we run x264 with the "veryslow" preset and VP8 with 
>> the "good, speed 0" setting in an attempt to produce comparable results.
>>
>> An overview of our results is available as follows:
>>
>> - A Quality comparison (psnr): 
>> http://downloads.webmproject.org/ietf_tests/vp8_vs_h264_quality.html
>>
>> - An Encode Speed comparison: 
>> http://downloads.webmproject.org/ietf_tests/vp8_vs_h264_speed.html
>>
>> - A comparison of the aggregate time required to decode all of the 
>> clips in the test: 
>> http://downloads.webmproject.org/ietf_tests/vp8vsh264-decodetime.txt
>>
>> All of our test scripts can either be downloaded from:
>> http://downloads.webmproject.org/ietf_tests/vp8_vs_h264.tar.xz
>> or checked out of our git/gerrit repository:
>> git clone http://git.chromium.org/webm/vpx_codec_comparison.git
>>
>> The file README.txt, contained within, presents details of how to 
>> build and run the tests.
>>
>> The compressed video files--the output from the quality tests--can 
>> also be downloaded:
>>
>> VP8:
>> http://downloads.webmproject.org/ietf_tests/vp8_videos 
>> <http://downloads.webmproject.org/ietf_tests/vp8_videos/>/index.html
>>
>> H.264:
>> http://downloads.webmproject.org/ietf_tests/h264_videos/index.html
>>
>> Adrian Grange
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> rtcweb mailing list
>> rtcweb@ietf.org
>> https://www.ietf.org/mailman/listinfo/rtcweb
>
>
>
> _______________________________________________
> rtcweb mailing list
> rtcweb@ietf.org
> https://www.ietf.org/mailman/listinfo/rtcweb