Re: [codec] draft test and processing plan for the IETF Codec

Ron <> Mon, 18 April 2011 11:47 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 43B34E0754 for <>; Mon, 18 Apr 2011 04:47:23 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 4LusBeEeC1Vm for <>; Mon, 18 Apr 2011 04:47:22 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id B0240E0737 for <>; Mon, 18 Apr 2011 04:47:20 -0700 (PDT)
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AvsEAEUhrE120qsf/2dsb2JhbAClWnjDNYVxBIVgiCI
Received: from (HELO audi.shelbyville.oz) ([]) by with ESMTP; 18 Apr 2011 21:17:18 +0930
Received: from localhost (localhost []) by audi.shelbyville.oz (Postfix) with ESMTP id 0290E4F8F3 for <>; Mon, 18 Apr 2011 21:17:17 +0930 (CST)
X-Virus-Scanned: Debian amavisd-new at audi.shelbyville.oz
Received: from audi.shelbyville.oz ([]) by localhost (audi.shelbyville.oz []) (amavisd-new, port 10024) with LMTP id HmpRxf-67nBp for <>; Mon, 18 Apr 2011 21:17:16 +0930 (CST)
Received: by audi.shelbyville.oz (Postfix, from userid 1000) id 5D1C84F8FE; Mon, 18 Apr 2011 21:17:16 +0930 (CST)
Date: Mon, 18 Apr 2011 21:17:16 +0930
From: Ron <>
Message-ID: <20110418114716.GC31013@audi.shelbyville.oz>
References: <> <> <>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <>
User-Agent: Mutt/1.5.20 (2009-06-14)
Subject: Re: [codec] draft test and processing plan for the IETF Codec
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 18 Apr 2011 11:47:23 -0000

Hi Anisse,

On Mon, Apr 18, 2011 at 09:52:42AM +0000, Anisse Taleb wrote:
> Hi Ben,
> > On 04/13/2011 03:32 AM, Anisse Taleb wrote:
> > > Please find attached a first draft of a test plan of the IETF codec
> > (Opus).
> > 
> > Thank you for drawing up this test plan, which clearly required a great
> > deal of thought.  The results of such testing would certainly be very
> > interesting to many.
> > 
> > However, I think the execution of such a test is clearly _not_ an
> > appropriate prerequisite for publishing a Proposed Standard.  By my
> > calculations, the draft plan presently calls for over 1300 hours of
> > listening tests, counting only audio being played, estimating 10-second
> > samples and the minimum number of listeners.  Even if many listeners are
> > listening in parallel, and overheads (such as delays between samples) are
> > low, conducting such a test would still take many months.
> It is always a good practice to first have a target on what would be tested
> and then find ways how to make the test realistic and reasonable. When it
> comes to the proposal itself, I think that shortcuts have been taken already.
> I am not against discussing the size of the test, the draft proposal was
> exactly made to initiate such discussion... 
> > 
> > Such an extensive, expensive battery of tests can hardly be justified on
> > some arbitrary codec version still under development.  
> I cannot agree more. Freeze a version of Opus, and let's check the quality of
> the codec. If it passes the quality expectations, it will become a standard. 
> -- But before that, clean up the code and the specification and fix the IPR
> issues. Right now the codec does not pass the "admin" part of requirements.

I thought it was already agreed that the people acting in good faith would
endeavour to conduct as much of this in parallel as possible.

I'm sure we have plenty of time to file off the rough edges while you gather
enough people to run the two hundred and something nonillion iterations of
your test that Gregory showed would be necessary for it to approach an even
remotely significant result that wasn't entirely a function of chance.

It saddens me to see you play a cheap shot like this at Ben, when so many
people are eagerly awaiting your explanation as to whether that was simply
an error in your math, or a factor you had not considered.  Or possibly an
essential ingredient in your insistence of a single do-or-die test?  That
nobody could possibly afford to repeat independently ...

So please, we've mapped both extremes of what a non-test might look like now,
and we've clearly shown, with absolute certainty, that this entire group will
never, not before the heat death of the universe, agree upon and perform one
single tell-all test which satisfies them all.  And that's before we consider
the users who aren't represented in this testing yet.

I think Cullen very accurately plotted where the middle ground may lay.
Let's all gravitate a little closer to that now, can we please?

Thanks Much!