Re: [codec] draft test and processing plan for the IETF Codec

[Paul Coverdale <[e-mail]>]

>
>I'm surprised we haven't seen a more intense reaction to this
>proposal yet.  Perhaps people are missing the less-than-obvious
>mathematical reality of it.
>
>If you have 10 requirements all of which must be met, where each is
>90% likely to be met, the chance of meeting all of them is 34.8%
>(.9^10).  The chance of failure increases exponentially with
>the number of requirements.
>
>This amplification effect is one reason why I've opposed additional
>requirements, even though I was quite confident that Opus was better
>than the competition.  Add enough requirements and Opus is sure to fail
>due to _chance_ no matter how good the codec is, even if the
>requirements each sound reasonable individually.
>
>In this case we have 162 requirements proposed. 75 "better than" (BT),
>and 87 "not worse than" (NWT), once you expand out all the loss rates,
>bit
>rates, etc.
>
>Moreover, because of measurement noise, Opus could meet all of the
>requirements and yet still fail some of the tests. Because there are
>so many requirements, even a small chance of false failure becomes
>significant.
>
>I did some rough numeric simulations with the tests proposed, using
>scores with a standard deviation of 1 (which is about what they were on
>the HA test), N = 144 as proposed, and Opus better than the
>comparison codec by 0.1. The chance of passing any single NWT
>requirement is then 0.9769, and the chance of passing any single BT
>requirement is 0.3802.
>
>The chance of passing all of them is
>0.9769^87 * 0.3802^75 = 4.1483e-33
>
>Which means about a 1 in 241 nonillion chance of passing all the tests,
>even assuming Opus actually met _all_ the stated requirements with a
>score +0.1 over the reference.
>e.
>
>This is so astronomically unlikely that I had to use an encyclopedia to
>find the name for the number.  I should have saved the time and just
>left it at "a farce".
>
>I urge the working group to keep this hazard in mind when considering
>the reasonableness of parallel MUST requirements on top of listening-
>test.

Greg,

I don't think the situation is as dire as you make out. Your analysis
assumes that all requirements are completely independent. This is not the
case, in many cases if you meet one requirement you are likely to meet
others of the same kind (eg performance as a function of bit rate).

But in any case, the statistical analysis procedure outlined in the test
plan doesn't assume that every requirement must be met with absolute
certainty, it allows for a confidence interval.

Regards,

...Paul