Re: [rtcweb] More H.264 vs VP8 tests

[Stefan Håkansson LK <[e-mail]>]

On 2013-06-25 17:36, John Koleszar wrote:
> On Sat, Jun 22, 2013 at 6:41 AM, Bo Burman <[e-mail]>
> wrote:
>> Hi all,
>>
>> We have had a look at Google's comparison between VP8 and H.264
>> constrained baseline that was posted on April 3rd
>> (http://www.ietf.org/mail-archive/web/rtcweb/current/msg07028.html).
>> This post contains, as the one mentioned above (and if the
>> attachments make it to the list), information on the exact tools
>> and options used for encoding and should thus be repeatable by
>> anyone interested.
>>
>> As was already stated by others on this list, one major problem is
>> that Google's test involves the rate control mechanism. Typically
>> codecs are measured with rate control turned off, since it acts as
>> a huge noise on the measurement. Instead we propose to compare the
>> codecs using fixed qp-levels. The qp-level is the quantization
>> parameter that affects the rate/distortion tradeoff. Comparing
>> using fixed qp-levels is what has been used when benchmarking HEVC
>> against H.264 in the JCT-VC standardization, for instance. We are
>> going to select a codec (essentially bit stream format), not a rate
>> control mechanism: Once the codec is selected you can choose
>> whatever rate control mechanism you wish.
>>
>> We used Google's excellent framework as the baseline and changed
>> the parameter settings in order to make it possible to measure
>> using fixed qp. We used the same sequences, but limited them to the
>> first 10 seconds since they varied from 10 seconds to minutes; this
>> also eased computation time.
>>
>> We used two H.264 encoder implementations: X264, which is an
>> open-source codec that can operate in everything from real-time to
>> slow, and JM which is the reference implementation that was used to
>> develop H.264. JM is very slow but attempts to be very efficient in
>> terms of bits per quality. The results were as follows:
>>
>> X264 baseline vs VP8: H.264 wins with 1% JM baseline vs VP8: H.264
>> wins with 4%
>>
>> Running times: X264: 1 hour 3 minutes VP8: 2 hours 0 minutes JM:
>> order of magnitude slower
>>
>> It is interesting to note that the measurements are more stable in
>> the new test; the variance of the percentages for the sequences is
>> now around 70, down from around 700 in Google's test of April 3rd.
>> We believe this is due to the removal of the rate controller, which
>> acts like noise on the measurements.
>>
>> We also tried setting H.264 to constrained high (no interlace and
>> no B-pictures, compared to high). The results were then:
>>
>> X264 constrained high vs VP8: H.264 wins with 25% JM constrained
>> high vs VP8: H.264 wins with 24%
>>
> [...]
>
> At one point, running the libvpx implementation of VP8 in fixed qp
> mode would effectively limit the encoder to only a single reference
> frame. I don't recall if it's still the case.

We have stepped through the decoding of the .webm files produced using 
our parameter settings and they indeed use several reference frames, so 
no, this seems not to be the case.

> Another unexpected
> behavior for people trying this mode is that making libvpx bump up
> against the lower quantizer without setting an approximately correct
> target bitrate can also put the encoder into its "over quant" mode,
> where it'll throw away additional residual trying to hit the target
> bitrate.

We have not seen this happen either. The target bitrate we set is 
ridiculously high so we do not see why the encoder should throw away 
anything that would cost bits, rather it would do the opposite.

> It doesn't have anything like the x264 ipratio (though I
> see you set this to 1, but this is something that bites people
> trying fixed-qp). This mode isn't really supported by libvpx, so I'd
> be careful about taking these results as representative of the very
> best VP8 can do. _______________________________________________
> rtcweb mailing list [e-mail]
> https://www.ietf.org/mailman/listinfo/rtcweb
>