Re: [video-codec] Charter issues from BoF - testing

["Cullen Jennings (fluffy)" <[e-mail]>]

On Nov 6, 2012, at 9:26 AM, Timothy B. Terriberry <[e-mail]> wrote:

>> 
>> 10. Have a clear idea how to get test results to inform WG decisions.
> 
> Fortunately, we're in a much better position with video than we were with audio. The objective metrics are more useful... everyone knows they're still flawed in various ways, but you can still make a lot of progress relying on such metrics (see the various ITU/MEPG efforts that rely on them exclusively). PSNR measured on a single, short clip comparing mostly similar algorithms actually correlate with human observer ratings pretty well [1]. Most comparisons between different codecs rely on them, too.
> 
> By contrast, in audio optimizing for SNR is actively harmful, and even more advanced metrics like PEAQ are essentially blind to things like transients, which are one of the most important source of artifacts for a transform codec. So if we wanted useful results, we didn't have a lot of options other than relying on human listening tests.
> 
> Humans are still the ultimate gold standard for video, of course. At least for the purposes of getting a technique adopted, I think we can just say the burden of proof lies on the person proposing the technique. If the visual improvement is large enough, even if the objective metrics say it looks worse, it shouldn't actually take much to convince people it's a good idea.

I'm pretty skeptical about PSNR but I was not really pushing on how we did testing here. I was more focused on how we will get any test data at all. 

Saying no feature goes in without test data in the charter would be a good way to address this. I'm sure there are others ways but what I want to avoid is a bunch of technical proposal which seem like they might be OK idea but actually don't gain much. We need some reasonable way to have enough information to decide if a feature is worth adding or not.