Re: [codec] draft test and processing plan for the IETF Codec

[Paul Coverdale <[e-mail]>]

As I mentioned earlier, the situation is not as bad as it may seem, certainly not a "1 in nonillion" chance of passing all requirements. Greg's analysis applies to flipping a coin that has a probability of .90 for "heads" and .10 for "tails." However, listener responses in a MOS test are not random - if they are, we throw that listener out of the results. The "randomness" that forms the basis of a statistical test derives from the distribution of responses ACROSS listeners rather than WITHIN listeners.
 
Regards,
 
...Paul
 
>-----Original Message-----
>From: Anisse Taleb [mailto:[e-mail]]
>Sent: Monday, April 18, 2011 6:37 PM
>To: Jean-Marc Valin; Paul Coverdale
>Cc: [e-mail]
>Subject: RE: [codec] draft test and processing plan for the IETF Codec
> 
>JM, Greg, Paul,
>[taking emails in chronological order was ill advised :-)]
> 
>I do not disagree with the statistical pitfalls you mention. As Paul
>stated and also what I wrote in a direct reply to this, there is no
>single uber-requirement to be passed by the codec, rather a vector of
>requirements that summarize the performance of the codec compared to
>other codecs. These have to be analyzed and discussed one by one.
> 
>Kind regards,
>/Anisse
> 
>> -----Original Message-----
>> From: [e-mail] [mailto:[e-mail]] On Behalf
>Of
>> Jean-Marc Valin
>> Sent: Thursday, April 14, 2011 3:07 PM
>> To: Paul Coverdale
>> Cc: [e-mail]
>> Subject: Re: [codec] draft test and processing plan for the IETF Codec
>> 
>> > I don't think the situation is as dire as you make out. Your
>analysis
>> > assumes that all requirements are completely independent. This is
>not the
>> > case, in many cases if you meet one requirement you are likely to
>meet
>> > others of the same kind (eg performance as a function of bit rate).
>> >
>> > But in any case, the statistical analysis procedure outlined in the
>test
>> > plan doesn't assume that every requirement must be met with absolute
>> > certainty, it allows for a confidence interval.
>> 
>> This is exactly what Greg is considering in his analysis. He's
>starting
>> from the assumption that the codec really meets *all* 162
>requirements.
>> Consider just the NWT requirements: if we were truly no worse than the
>> reference codec, then with 87 tests against a 95% confidence interval,
>we
>> would be expected to fail about 4 tests just by random chance.
>Considering
>> both NWT and BT requirements, the odds of passing Anisse's proposed
>test
>> plan given the assumptions above are 4.1483e-33. See
>http://xkcd.com/882/
>> for a more rigorous analysis.
>> 
>> Cheers,
>> 
>>    Jean-Marc