Re: [codec] draft test and processing plan for the IETF Codec

[Ron <[e-mail]>]

Hi Anisse,

On Mon, Apr 18, 2011 at 09:52:42AM +0000, Anisse Taleb wrote:
> Hi Ben,
> 
> > On 04/13/2011 03:32 AM, Anisse Taleb wrote:
> > > Please find attached a first draft of a test plan of the IETF codec
> > (Opus).
> > 
> > Thank you for drawing up this test plan, which clearly required a great
> > deal of thought.  The results of such testing would certainly be very
> > interesting to many.
> > 
> > However, I think the execution of such a test is clearly _not_ an
> > appropriate prerequisite for publishing a Proposed Standard.  By my
> > calculations, the draft plan presently calls for over 1300 hours of
> > listening tests, counting only audio being played, estimating 10-second
> > samples and the minimum number of listeners.  Even if many listeners are
> > listening in parallel, and overheads (such as delays between samples) are
> > low, conducting such a test would still take many months.
> 
> It is always a good practice to first have a target on what would be tested
> and then find ways how to make the test realistic and reasonable. When it
> comes to the proposal itself, I think that shortcuts have been taken already.
> I am not against discussing the size of the test, the draft proposal was
> exactly made to initiate such discussion... 
> > 
> > Such an extensive, expensive battery of tests can hardly be justified on
> > some arbitrary codec version still under development.  
> 
> I cannot agree more. Freeze a version of Opus, and let's check the quality of
> the codec. If it passes the quality expectations, it will become a standard. 
> 
> -- But before that, clean up the code and the specification and fix the IPR
> issues. Right now the codec does not pass the "admin" part of requirements.

I thought it was already agreed that the people acting in good faith would
endeavour to conduct as much of this in parallel as possible.

I'm sure we have plenty of time to file off the rough edges while you gather
enough people to run the two hundred and something nonillion iterations of
your test that Gregory showed would be necessary for it to approach an even
remotely significant result that wasn't entirely a function of chance.

It saddens me to see you play a cheap shot like this at Ben, when so many
people are eagerly awaiting your explanation as to whether that was simply
an error in your math, or a factor you had not considered.  Or possibly an
essential ingredient in your insistence of a single do-or-die test?  That
nobody could possibly afford to repeat independently ...


So please, we've mapped both extremes of what a non-test might look like now,
and we've clearly shown, with absolute certainty, that this entire group will
never, not before the heat death of the universe, agree upon and perform one
single tell-all test which satisfies them all.  And that's before we consider
the users who aren't represented in this testing yet.

I think Cullen very accurately plotted where the middle ground may lay.
Let's all gravitate a little closer to that now, can we please?

Thanks Much!
Ron