Re: [rtcweb] More H.264 vs VP8 tests

[Saverio Mascolo <[e-mail]>]

an evaluation of how h264 and vp8 are able to match a changing quality
would be interesting


On Sat, Jun 29, 2013 at 3:17 PM, Stefan Håkansson LK <
[e-mail]> wrote:

> On 6/25/13 12:17 PM, Harald Alvestrand wrote:
> > Again - thanks for releasing this openly!
> >
> > I ran the scripts (with a few tweaks; you run on a system where sh is
> > bash, not dash, for instance), and got the same numbers within +/- 0.5%
> > (probably some binary version skew);
>
> I re-run the scripts on another computer with another OS today, and I
> get exactly the same results as Bo sent out. I noted however that if the
> input clips are not cut at 10s (but used in their entire length) the
> results get slightly different, but within +/- 0.5%. Can this be the
> reason why you get slightly different numbers?
>
>
> > we may have disagreements on the
> > parameters to use, but we agree on the numbers those parameters produce.
> >
> > (I have since modified the Google framework to include a script that
> > pulls in the sources for the needed binaries and compiles them - if you
> > want to make 100% sure people are working from the same sources, you may
> > want to rebase to a newer version of the comparision toolkit.)
> >
> > On 06/22/2013 03:41 PM, Bo Burman wrote:
> >> Hi all,
> >>
> >> We have had a look at Google's comparison between VP8 and H.264
> constrained baseline that was posted on April 3rd (
> http://www.ietf.org/mail-archive/web/rtcweb/current/msg07028.html). This
> post contains, as the one mentioned above (and if the attachments make it
> to the list), information on the exact tools and options used for encoding
> and should thus be repeatable by anyone interested.
> >>
> >> As was already stated by others on this list, one major problem is that
> Google's test involves the rate control mechanism. Typically codecs are
> measured with rate control turned off, since it acts as a huge noise on the
> measurement. Instead we propose to compare the codecs using fixed
> qp-levels. The qp-level is the quantization parameter that affects the
> rate/distortion tradeoff. Comparing using fixed qp-levels is what has been
> used when benchmarking HEVC against H.264 in the JCT-VC standardization,
> for instance. We are going to select a codec (essentially bit stream
> format), not a rate control mechanism: Once the codec is selected you can
> choose whatever rate control mechanism you wish.
> >>
> >> We used Google's excellent framework as the baseline and changed the
> parameter settings in order to make it possible to measure using fixed qp.
> We used the same sequences, but limited them to the first 10 seconds since
> they varied from 10 seconds to minutes; this also eased computation time.
> >>
> >> We used two H.264 encoder implementations: X264, which is an
> open-source codec that can operate in everything from real-time to slow,
> and JM which is the reference implementation that was used to develop
> H.264. JM is very slow but attempts to be very efficient in terms of bits
> per quality. The results were as follows:
> >>
> >> X264 baseline vs VP8: H.264 wins with 1%
> >> JM baseline vs VP8: H.264 wins with 4%
> >>
> >> Running times:
> >> X264: 1 hour 3 minutes
> >> VP8: 2 hours 0 minutes
> >> JM: order of magnitude slower
> >>
> >> It is interesting to note that the measurements are more stable in the
> new test; the variance of the percentages for the sequences is now around
> 70, down from around 700 in Google's test of April 3rd.  We believe this is
> due to the removal of the rate controller, which acts like noise on the
> measurements.
> >>
> >> We also tried setting H.264 to constrained high (no interlace and no
> B-pictures, compared to high). The results were then:
> >>
> >> X264 constrained high vs VP8: H.264 wins with 25%
> >> JM constrained high vs VP8: H.264 wins with 24%
> >>
> >> We also note that the script that Google provided to calculate the rate
> differences ("BD-rate") does not give exactly the same numbers as the
> JCT-VC-way of calculating BD-rate. The main difference is that the JM score
> for constrained high is better (around 29%) if the JCT-VC way of
> calculating BD-rate is used.
> >>
> >> In summary we think that proper testing can conclude that there is no
> clear performance advantage to any codec between VP8 and H.264 baseline.
> When comparing VP8 against H.264 constrained high on the other hand, it
> seems like there is an advantage for H.264 constrained high.
> >>
> >> The attached file includes the files necessary to reproduce the test.
> >>
> >> Best Regards,
> >>
> >> Bo Burman
> >>
> >>
> >> _______________________________________________
> >> rtcweb mailing list
> >> [e-mail]
> >> https://www.ietf.org/mailman/listinfo/rtcweb
> >
>
> _______________________________________________
> rtcweb mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/rtcweb
>