Re: [codec] draft test and processing plan for the IETF Codec

Paul Coverdale <> Tue, 19 April 2011 02:20 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id D3E67E0684 for <>; Mon, 18 Apr 2011 19:20:34 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.598
X-Spam-Status: No, score=-2.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id OQ5KwjRDqKhR for <>; Mon, 18 Apr 2011 19:20:33 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id 6F9DBE0611 for <>; Mon, 18 Apr 2011 19:20:33 -0700 (PDT)
Received: from COL103-W39 ([]) by with Microsoft SMTPSVC(6.0.3790.4675); Mon, 18 Apr 2011 19:20:32 -0700
Message-ID: <COL103-W39BD44E16F636F4DB06934D0900@phx.gbl>
Content-Type: multipart/alternative; boundary="_baf0e3cb-a6a0-41f8-b175-933c04d4f169_"
X-Originating-IP: []
From: Paul Coverdale <>
Date: Tue, 19 Apr 2011 02:20:32 +0000
Importance: Normal
MIME-Version: 1.0
X-OriginalArrivalTime: 19 Apr 2011 02:20:32.0811 (UTC) FILETIME=[5C2A97B0:01CBFE38]
Subject: Re: [codec] draft test and processing plan for the IETF Codec
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Tue, 19 Apr 2011 02:20:35 -0000

As I mentioned earlier, the situation is not as bad as it may seem, certainly not a "1 in nonillion" chance of passing all requirements. Greg's analysis applies to flipping a coin that has a probability of .90 for "heads" and .10 for "tails." However, listener responses in a MOS test are not random - if they are, we throw that listener out of the results. The "randomness" that forms the basis of a statistical test derives from the distribution of responses ACROSS listeners rather than WITHIN listeners.
>-----Original Message-----
>From: Anisse Taleb []
>Sent: Monday, April 18, 2011 6:37 PM
>To: Jean-Marc Valin; Paul Coverdale
>Subject: RE: [codec] draft test and processing plan for the IETF Codec
>JM, Greg, Paul,
>[taking emails in chronological order was ill advised :-)]
>I do not disagree with the statistical pitfalls you mention. As Paul
>stated and also what I wrote in a direct reply to this, there is no
>single uber-requirement to be passed by the codec, rather a vector of
>requirements that summarize the performance of the codec compared to
>other codecs. These have to be analyzed and discussed one by one.
>Kind regards,
>> -----Original Message-----
>> From: [] On Behalf
>> Jean-Marc Valin
>> Sent: Thursday, April 14, 2011 3:07 PM
>> To: Paul Coverdale
>> Cc:
>> Subject: Re: [codec] draft test and processing plan for the IETF Codec
>> > I don't think the situation is as dire as you make out. Your
>> > assumes that all requirements are completely independent. This is
>not the
>> > case, in many cases if you meet one requirement you are likely to
>> > others of the same kind (eg performance as a function of bit rate).
>> >
>> > But in any case, the statistical analysis procedure outlined in the
>> > plan doesn't assume that every requirement must be met with absolute
>> > certainty, it allows for a confidence interval.
>> This is exactly what Greg is considering in his analysis. He's
>> from the assumption that the codec really meets *all* 162
>> Consider just the NWT requirements: if we were truly no worse than the
>> reference codec, then with 87 tests against a 95% confidence interval,
>> would be expected to fail about 4 tests just by random chance.
>> both NWT and BT requirements, the odds of passing Anisse's proposed
>> plan given the assumptions above are 4.1483e-33. See
>> for a more rigorous analysis.
>> Cheers,
>>    Jean-Marc