Re: [codec] draft test and processing plan for the IETF Codec

[Jean-Marc Valin <[e-mail]>]

> I don't think the situation is as dire as you make out. Your analysis
> assumes that all requirements are completely independent. This is not the
> case, in many cases if you meet one requirement you are likely to meet
> others of the same kind (eg performance as a function of bit rate).
>
> But in any case, the statistical analysis procedure outlined in the test
> plan doesn't assume that every requirement must be met with absolute
> certainty, it allows for a confidence interval.

This is exactly what Greg is considering in his analysis. He's starting 
from the assumption that the codec really meets *all* 162 requirements. 
Consider just the NWT requirements: if we were truly no worse than the 
reference codec, then with 87 tests against a 95% confidence interval, we 
would be expected to fail about 4 tests just by random chance. Considering 
both NWT and BT requirements, the odds of passing Anisse's proposed test 
plan given the assumptions above are 4.1483e-33. See http://xkcd.com/882/ 
for a more rigorous analysis.

Cheers,

	Jean-Marc