Re: [rtcweb] VP8 vs H.264

It is really difficult to at all comment on this since there is far too little data available. There is no way to review the settings and parameters used, and of course no way to repeat the tests, and hopefully our previous inputs related to objective quality evaluation has made it clear to everyone that parameter settings make a huge difference.

It can also be noted that even if the results are assumed to be correct (which there is currently no way to verify) the conclusions drawn seem incorrect. E.g., it is stated that "In 7/10 sequences tested VP8 LD was clearly better than AVC constrained baseline", but when examining the plots the confidence intervals seem to be overlapping.

Without any more information it can only be concluded that this test has very little value.

> -----Original Message-----
> From: rtcweb-bounces@ietf.org [mailto:rtcweb-bounces@ietf.org] On Behalf Of Basil Mohamed Gohar
> Sent: den 23 oktober 2013 15:32
> To: rtcweb@ietf.org
> Subject: Re: [rtcweb] VP8 vs H.264 - subjective evaluation
> 
> On 10/23/2013 04:26 AM, Harald Alvestrand wrote:
> > On 10/23/2013 02:40 AM, Basil Mohamed Gohar wrote:
> >> On 10/22/2013 06:24 AM, Harald Alvestrand wrote:
> >>> In my VP8 advocate draft, I referred to a subjective evaluation test
> >>> (test with actual human viewers) done between VP8 and H.264 Baseline
> >>> by a neutral
> >>> (non-Google) laboratory.
> >>>
> >>> The attached presentation is the writeup of the results of that test.
> >>>
> >> Are the sources used in the video comparison test available anywhere?
> >>
> > They are under an MPEG license, so they're freely available to MPEG
> > members for use in developing MPEG standards. If you fall within that
> > category, I can send you the FTP server and password.
> >
> > MPEG doesn't have the tradition of open materials that the IETF has.
> >
> 
> Well, that's unfortunate.  Thanks for clarifying, though.
> 
> It would be nice if, when time permits, a similar test can be made, using the same criteria (encoding parameters, etc.),
> but with known clips (e.g., from derf's collection, which has everything, including both SD and HD clips).  Publishing the
> versions and exact encoding parameters (including command line usage) would be helpful to that aim, I suppose.
> 
> --
> Libre Video
> http://librevideo.org
> _______________________________________________
> rtcweb mailing list
> rtcweb@ietf.org
> https://www.ietf.org/mailman/listinfo/rtcweb