Re: [codec] draft test and processing plan for the IETF Codec

[Anisse Taleb <[e-mail]>]

Paul,

Strictly speaking, the probability of failing at least one requirement increases (or stays constant in dependent cases) with increasing the number of requirements. Of course as you mention, the responses of the listeners are not really random and a heads-tail modeling of the pass-fail is way too simplistic. I didn't look at the analysis in depth, neither did I verify the numbers.
I think Greg is mostly concerned with the size of the test and how the analysis of the requirements would be used to derive a conclusion about the codec.



[Off topic]
Statistics are quite fun to play with, a while ago a Conspiracy theorist tried to convince me that man never landed on the moon and that it was impossible that Apollo 11 made it. The millions of components, given the technology of that time, had  a significant probability of failure that in total it was beyond doubt that such a rocket would have gone off course.

Kind regards,
/Anisse


From: [e-mail] [mailto:[e-mail]] On Behalf Of Paul Coverdale
Sent: Tuesday, April 19, 2011 4:21 AM
To: [e-mail]
Subject: Re: [codec] draft test and processing plan for the IETF Codec

As I mentioned earlier, the situation is not as bad as it may seem, certainly not a "1 in nonillion" chance of passing all requirements. Greg's analysis applies to flipping a coin that has a probability of .90 for "heads" and .10 for "tails." However, listener responses in a MOS test are not random - if they are, we throw that listener out of the results. The "randomness" that forms the basis of a statistical test derives from the distribution of responses ACROSS listeners rather than WITHIN listeners.

Regards,

...Paul

>-----Original Message-----
>From: Anisse Taleb [mailto:[e-mail]]<mailto:[mailto:[e-mail]]>
>Sent: Monday, April 18, 2011 6:37 PM
>To: Jean-Marc Valin; Paul Coverdale
>Cc: [e-mail]<mailto:[e-mail]>
>Subject: RE: [codec] draft test and processing plan for the IETF Codec
>
>JM, Greg, Paul,
>[taking emails in chronological order was ill advised :-)]
>
>I do not disagree with the statistical pitfalls you mention. As Paul
>stated and also what I wrote in a direct reply to this, there is no
>single uber-requirement to be passed by the codec, rather a vector of
>requirements that summarize the performance of the codec compared to
>other codecs. These have to be analyzed and discussed one by one.
>
>Kind regards,
>/Anisse
>
>> -----Original Message-----
>> From: [e-mail]<mailto:[e-mail]> [mailto:[e-mail]]<mailto:[mailto:[e-mail]]> On Behalf
>Of
>> Jean-Marc Valin
>> Sent: Thursday, April 14, 2011 3:07 PM
>> To: Paul Coverdale
>> Cc: [e-mail]<mailto:[e-mail]>
>> Subject: Re: [codec] draft test and processing plan for the IETF Codec
>>
>> > I don't think the situation is as dire as you make out. Your
>analysis
>> > assumes that all requirements are completely independent. This is
>not the
>> > case, in many cases if you meet one requirement you are likely to
>meet
>> > others of the same kind (eg performance as a function of bit rate).
>> >
>> > But in any case, the statistical analysis procedure outlined in the
>test
>> > plan doesn't assume that every requirement must be met with absolute
>> > certainty, it allows for a confidence interval.
>>
>> This is exactly what Greg is considering in his analysis. He's
>starting
>> from the assumption that the codec really meets *all* 162
>requirements.
>> Consider just the NWT requirements: if we were truly no worse than the
>> reference codec, then with 87 tests against a 95% confidence interval,
>we
>> would be expected to fail about 4 tests just by random chance.
>Considering
>> both NWT and BT requirements, the odds of passing Anisse's proposed
>test
>> plan given the assumptions above are 4.1483e-33. See
>http://xkcd.com/882/
>> for a more rigorous analysis.
>>
>> Cheers,
>>
>>   Jean-Marc