Re: [rtcweb] New VP8 vs H.264 tests uploaded

["Mo Zanaty (mzanaty)" <[e-mail]>]

Realtime/low latency and constrained bitrate are obviously important for the actual implementation used. Thomas was pointing out that these factors have nothing to do with the codec technology itself, since they are purely encoder implementation optimizations. There is nothing in the VP8 or H.264 standard that uniquely provides realtime/low latency or constrained bitrate. Those are attributes of encoder implementations which are not part of the standard.

So the question was whether we care about evaluating codec technology or specific implementations. If the former, then tests should be staged in the same way codec experts evaluate codec technology/tools. If the latter, then tests should be staged using the target implementations.

I'm not aware of conferencing applications which use x264, because it was designed and optimized for transcoding (dvd rips to blu-ray) not conferencing. Most importantly, x264 cbr mode is inappropriate for conferencing since it is for broadcast MPEG transport streams that must be absolutely CBR to avoid M2TS-mux overflow or underflow, and it will actually insert filler data instead of real frame data to hit the CBR rate exactly. Looking at the results which show the worst H.264 bitrate (62% above VP8) in gipsrecstat_1280_720_50_1485kbps.mkv, there is almost as much filler data as real frame data, meaning the true bitrate of real frame data is almost half what is reported in the results. (See attached if it makes it through.)

While the results are bad, the methodology, effort and transparency are very good (if we want to compare implementations not standards). I can rerun without the bogus fillers and post the results next week, unless someone else can do it faster. But as Thomas pointed out, the technologies themselves are comparable as far as coding tools, so any results which show significant differences are either suspect or explained by differences in encoder implementations or settings not the codec technology itself.

Mo


From: [e-mail] [mailto:[e-mail]] On Behalf Of Leon Geyser
Sent: Thursday, April 04, 2013 12:56 PM
To: Thomas Davies (thdavies)
Cc: [e-mail]
Subject: Re: [rtcweb] New VP8 vs H.264 tests uploaded

If the purpose is to show whether vp8 is superior as a *technology* to h264 CBP, then I think the comparison should use the best settings you have (ideally with a special full-on non-real time implementation) and test against the JM reference encoder. Ideally you would use the same or similar GOP structures, number of references, prediction and QP hierarchies.
I thought WebRTC was meant for real-time communication. What would it benefit us if we test settings that won't be used or can't be used in practice?

The tests need to test the encoders at realtime/low latency and at a constrained bitrate mode like CBR. We aren't archiving videos here :)

A graph that shows the bitrate over time for each clip could be usefull to make sure that no encoder spikes the bitrate too high at certain moments.
I welcome changes to the encoder settings as long as they stay realtime/low latency and constrained bitrate.

On 4 April 2013 17:05, Thomas Davies <[e-mail]<mailto:[e-mail]>> wrote:
Harald,

I think there are quite a few problems with the comparison you have posted.

1. Looking at the sequences there is a very major difference between the initial intra frame qualities. When I encode just one frame of sequence gipsrecomotion using the parameters in the script at 1Mb/s then the intra frame is 3 times larger with vp8 than with x264.

With video conferencing content, the quality of the initial I frame has a big impact that can last for many seconds - certainly the length of these clips. You can easily get gains by increasing the quality difference between an I frame and subsequent frames.

x264 seems to have a policy of initially undershooting the bitrate substantially and ramping up, whereas vpxenc has a different approach. During this ramp up period the quality is very much worse. I can't find a way to persuade x264 to behave differently.

This is a good illustration of why including rate control in comparisons is a bad idea.

2. Likewise, looking at the individual frame sizes, it seems vpxenc is using a quality hierarchy with a length of 8 ("hiercharchical-P") where every 8th frame is about 4x bigger than the others. x264 has a constant target per frame.

Hierarchical P frames are a really good idea, and can easily get you 10-20% gain with a big separation like this, at a cost in latency. Again I don't know how to make x264 do this, but the technique is applicable to any codec and is used in the JM reference.

3. The x264 settings are a bit of a black art, but appear not to be ideal after all. I am definitely no expert but I found that when encoding gipsrecomotion at 1Mb/s:

- setting --threads 1 improves quality by a full 1dB (vpxenc seems to run single threaded by default)
- reducing the number of references from 3 to 2 (--ref 2) reduces the load very substantially at very little loss (0.2dB or so).

So with --threads 1 --ref 2, I found x264 ran more than 2x faster than vpxenc for this data point and had much better quality than before. vpxenc is still better (about 1dB), but very possibly within the range of hierarchical P coding improvements.

Incidentally, I don't think that x264 performs particularly well at these high complexity settings, at least for video conferencing, no doubt as other more practical settings have been targeted. x264 appears to have a quality ceiling that the JM does not have.

4. Another (smaller) issue is that the reported PSNR is combined luma and chroma over all frames. It's relatively easy to improve chroma PSNR at a small cost in bits, and usually it is best to ignore chroma PSNR or (possibly) give it a small weight. The arithmetic mean of frame PSNRs is generally used rather than the PSNR of the whole sequence, also. I would very much like separate component PSNRs in tests. The figures I quote above are luma PSNR.

If the purpose is to show whether vp8 is superior as a *technology* to h264 CBP, then I think the comparison should use the best settings you have (ideally with a special full-on non-real time implementation) and test against the JM reference encoder. Ideally you would use the same or similar GOP structures, number of references, prediction and QP hierarchies.

Comparing different real-time implementations of different codecs trying to do high quality coding with different GOP structures and using rate control with different strategies is just a waste of time. The first two elements in the list above are alone worth a very significant amount of bit rate.

On the other hand, a quick perusal of the actual tools would suggest that vp8 and h264 CBP are likely "comparable" and the variation between implementations of the same technology would be bigger than the variation between the technologies. If we could agree that then a lot of time could be saved.

best regards

Thomas




On 04/04/13 09:53, Sergio Garcia Murillo wrote:
Hi Adrian,

Could you explain how the encoding parametrization is comparable?

x264 --nal-hrd cbr --vbv-maxrate ${rate} --vbv-bufsize ${rate} \
      --vbv-init 0.8 --bitrate ${rate} --fps ${frame_rate} \
      --profile baseline --no-scenecut --keyint infinite --preset veryslow \
      --input-res ${width}x${height} \
      --tune psnr \
      -o ./encoded_clips/h264/${clip_stem}_${rate}kbps.mkv ${filename} \
      2> ./logs/h264/${clip_stem}_${rate}kbps.txt

vs:

 ./bin/vpxenc --lag-in-frames=0 --target-bitrate=${rate} --kf-min-dist=3000 \
      --kf-max-dist=3000 --cpu-used=0 --fps=${frame_rate}/1 --static-thresh=0 \
      --token-parts=1 --drop-frame=0 --end-usage=cbr --min-q=2 --max-q=56 \
      --undershoot-pct=100 --overshoot-pct=15 --buf-sz=1000 \
      --buf-initial-sz=800 --buf-optimal-sz=1000 --max-intra-rate=1200 \
      --resize-allowed=0 --drop-frame=0 --passes=1 --good --noise-sensitivity=0 \
      -w ${width} -h ${height} ${filename} --codec=vp8 \
      -o ./encoded_clips/vp8/${clip_stem}_${rate}kbps.webm \
      &>./logs/vp8/${clip_stem}_${rate}kbps.txt

Best regards
Sergio

El 03/04/2013 18:20, Adrian Grange escribió:
We have uploaded a new set of test results comparing VP8 to H.264. This latest set contains fixes for some of the problems in the previous set. We would like to extend our thanks to those who made suggestions as to how we could improve our methodology and encourage suggestions as to how we can make further improvements.

In these tests we run x264 with the "veryslow" preset and VP8 with the "good, speed 0" setting in an attempt to produce comparable results.

An overview of our results is available as follows:

- A Quality comparison (psnr): http://downloads.webmproject.org/ietf_tests/vp8_vs_h264_quality.html

- An Encode Speed comparison: http://downloads.webmproject.org/ietf_tests/vp8_vs_h264_speed.html

- A comparison of the aggregate time required to decode all of the clips in the test: http://downloads.webmproject.org/ietf_tests/vp8vsh264-decodetime.txt

All of our test scripts can either be downloaded from:
http://downloads.webmproject.org/ietf_tests/vp8_vs_h264.tar.xz
or checked out of our git/gerrit repository:
git clone http://git.chromium.org/webm/vpx_codec_comparison.git

The file README.txt, contained within, presents details of how to build and run the tests.

The compressed video files--the output from the quality tests--can also be downloaded:

VP8:
http://downloads.webmproject.org/ietf_tests/vp8_videos<http://downloads.webmproject.org/ietf_tests/vp8_videos/>/index.html

H.264:
http://downloads.webmproject.org/ietf_tests/h264_videos/index.html

Adrian Grange







_______________________________________________

rtcweb mailing list

[e-mail]<mailto:[e-mail]>

https://www.ietf.org/mailman/listinfo/rtcweb



_______________________________________________

rtcweb mailing list

[e-mail]<mailto:[e-mail]>

https://www.ietf.org/mailman/listinfo/rtcweb


_______________________________________________
rtcweb mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/rtcweb