Re: [rtcweb] New VP8 vs H.264 tests uploaded

[Leon Geyser <[e-mail]>]

>
> If the purpose is to show whether vp8 is superior as a *technology* to
> h264 CBP, then I think the comparison should use the best settings you have
> (ideally with a special full-on non-real time implementation) and test
> against the JM reference encoder. Ideally you would use the same or similar
> GOP structures, number of references, prediction and QP hierarchies.
>
I thought WebRTC was meant for real-time communication. What would it
benefit us if we test settings that won't be used or can't be used in
practice?

The tests need to test the encoders at realtime/low latency and at a
constrained bitrate mode like CBR. We aren't archiving videos here :)

A graph that shows the bitrate over time for each clip could be usefull to
make sure that no encoder spikes the bitrate too high at certain moments.
I welcome changes to the encoder settings as long as they stay realtime/low
latency and constrained bitrate.

On 4 April 2013 17:05, Thomas Davies <[e-mail]> wrote:

>  Harald,
>
> I think there are quite a few problems with the comparison you have posted.
>
> 1. Looking at the sequences there is a very major difference between the
> initial intra frame qualities. When I encode just one frame of sequence
> gipsrecomotion using the parameters in the script at 1Mb/s then the intra
> frame is 3 times larger with vp8 than with x264.
>
> With video conferencing content, the quality of the initial I frame has a
> big impact that can last for many seconds - certainly the length of these
> clips. You can easily get gains by increasing the quality difference
> between an I frame and subsequent frames.
>
> x264 seems to have a policy of initially undershooting the bitrate
> substantially and ramping up, whereas vpxenc has a different approach.
> During this ramp up period the quality is very much worse. I can't find a
> way to persuade x264 to behave differently.
>
> This is a good illustration of why including rate control in comparisons
> is a bad idea.
>
> 2. Likewise, looking at the individual frame sizes, it seems vpxenc is
> using a quality hierarchy with a length of 8 ("hiercharchical-P") where
> every 8th frame is about 4x bigger than the others. x264 has a constant
> target per frame.
>
> Hierarchical P frames are a really good idea, and can easily get you
> 10-20% gain with a big separation like this, at a cost in latency. Again I
> don't know how to make x264 do this, but the technique is applicable to any
> codec and is used in the JM reference.
>
> 3. The x264 settings are a bit of a black art, but appear not to be ideal
> after all. I am definitely no expert but I found that when encoding
> gipsrecomotion at 1Mb/s:
>
> - setting --threads 1 improves quality by a full 1dB (vpxenc seems to run
> single threaded by default)
> - reducing the number of references from 3 to 2 (--ref 2) reduces the load
> very substantially at very little loss (0.2dB or so).
>
> So with --threads 1 --ref 2, I found x264 ran more than 2x faster than
> vpxenc for this data point and had much better quality than before. vpxenc
> is still better (about 1dB), but very possibly within the range of
> hierarchical P coding improvements.
>
> Incidentally, I don't think that x264 performs particularly well at these
> high complexity settings, at least for video conferencing, no doubt as
> other more practical settings have been targeted. x264 appears to have a
> quality ceiling that the JM does not have.
>
> 4. Another (smaller) issue is that the reported PSNR is combined luma and
> chroma over all frames. It's relatively easy to improve chroma PSNR at a
> small cost in bits, and usually it is best to ignore chroma PSNR or
> (possibly) give it a small weight. The arithmetic mean of frame PSNRs is
> generally used rather than the PSNR of the whole sequence, also. I would
> very much like separate component PSNRs in tests. The figures I quote above
> are luma PSNR.
>
> If the purpose is to show whether vp8 is superior as a *technology* to
> h264 CBP, then I think the comparison should use the best settings you have
> (ideally with a special full-on non-real time implementation) and test
> against the JM reference encoder. Ideally you would use the same or similar
> GOP structures, number of references, prediction and QP hierarchies.
>
> Comparing different real-time implementations of different codecs trying
> to do high quality coding with different GOP structures and using rate
> control with different strategies is just a waste of time. The first two
> elements in the list above are alone worth a very significant amount of bit
> rate.
>
> On the other hand, a quick perusal of the actual tools would suggest that
> vp8 and h264 CBP are likely "comparable" and the variation between
> implementations of the same technology would be bigger than the variation
> between the technologies. If we could agree that then a lot of time could
> be saved.
>
> best regards
>
> Thomas
>
>
>
>
> On 04/04/13 09:53, Sergio Garcia Murillo wrote:
>
> Hi Adrian,
>
> Could you explain how the encoding parametrization is comparable?
>
> x264 --nal-hrd cbr --vbv-maxrate ${rate} --vbv-bufsize ${rate} \
>       --vbv-init 0.8 --bitrate ${rate} --fps ${frame_rate} \
>       --profile baseline --no-scenecut --keyint infinite --preset veryslow
> \
>       --input-res ${width}x${height} \
>       --tune psnr \
>       -o ./encoded_clips/h264/${clip_stem}_${rate}kbps.mkv ${filename} \
>       2> ./logs/h264/${clip_stem}_${rate}kbps.txt
>
> vs:
>
>  ./bin/vpxenc --lag-in-frames=0 --target-bitrate=${rate}
> --kf-min-dist=3000 \
>       --kf-max-dist=3000 --cpu-used=0 --fps=${frame_rate}/1
> --static-thresh=0 \
>       --token-parts=1 --drop-frame=0 --end-usage=cbr --min-q=2 --max-q=56
> \
>       --undershoot-pct=100 --overshoot-pct=15 --buf-sz=1000 \
>       --buf-initial-sz=800 --buf-optimal-sz=1000 --max-intra-rate=1200 \
>       --resize-allowed=0 --drop-frame=0 --passes=1 --good
> --noise-sensitivity=0 \
>       -w ${width} -h ${height} ${filename} --codec=vp8 \
>       -o ./encoded_clips/vp8/${clip_stem}_${rate}kbps.webm \
>       &>./logs/vp8/${clip_stem}_${rate}kbps.txt
>
> Best regards
> Sergio
>
> El 03/04/2013 18:20, Adrian Grange escribió:
>
> We have uploaded a new set of test results comparing VP8 to H.264. This
> latest set contains fixes for some of the problems in the previous set. We
> would like to extend our thanks to those who made suggestions as to how we
> could improve our methodology and encourage suggestions as to how we can
> make further improvements.
>
>  In these tests we run x264 with the "veryslow" preset and VP8 with the
> "good, speed 0" setting in an attempt to produce comparable results.
>
>  An overview of our results is available as follows:
>
>  - A Quality comparison (psnr):
> http://downloads.webmproject.org/ietf_tests/vp8_vs_h264_quality.html
>
>  - An Encode Speed comparison:
> http://downloads.webmproject.org/ietf_tests/vp8_vs_h264_speed.html
>
>  - A comparison of the aggregate time required to decode all of the clips
> in the test:
> http://downloads.webmproject.org/ietf_tests/vp8vsh264-decodetime.txt
>
>  All of our test scripts can either be downloaded from:
>   http://downloads.webmproject.org/ietf_tests/vp8_vs_h264.tar.xz
>  or checked out of our git/gerrit repository:
> git clone http://git.chromium.org/webm/vpx_codec_comparison.git
>
>  The file README.txt, contained within, presents details of how to build
> and run the tests.
>
>  The compressed video files--the output from the quality tests--can also
> be downloaded:
>
>  VP8:
> http://downloads.webmproject.org/ietf_tests/vp8_videos/index.html
>
>  H.264:
> http://downloads.webmproject.org/ietf_tests/h264_videos/index.html
>
>  Adrian Grange
>
>
>
>
>
>
>
> _______________________________________________
> rtcweb mailing [e-mail]://www.ietf.org/mailman/listinfo/rtcweb
>
>
>
>
> _______________________________________________
> rtcweb mailing [e-mail]://www.ietf.org/mailman/listinfo/rtcweb
>
>
>
>
> _______________________________________________
> rtcweb mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/rtcweb
>
>