Re: [codec] draft test and processing plan for the IETF Codec

[Anisse Taleb <[e-mail]>]

Hi Ben,

> On 04/13/2011 03:32 AM, Anisse Taleb wrote:
> > Please find attached a first draft of a test plan of the IETF codec
> (Opus).
> 
> Thank you for drawing up this test plan, which clearly required a great
> deal of thought.  The results of such testing would certainly be very
> interesting to many.
> 
> However, I think the execution of such a test is clearly _not_ an
> appropriate prerequisite for publishing a Proposed Standard.  By my
> calculations, the draft plan presently calls for over 1300 hours of
> listening tests, counting only audio being played, estimating 10-second
> samples and the minimum number of listeners.  Even if many listeners are
> listening in parallel, and overheads (such as delays between samples) are
> low, conducting such a test would still take many months.

It is always a good practice to first have a target on what would be tested and then find ways how to make the test realistic and reasonable. When it comes to the proposal itself, I think that shortcuts have been taken already. I am not against discussing the size of the test, the draft proposal was exactly made to initiate such discussion... 
> 
> Such an extensive, expensive battery of tests can hardly be justified on
> some arbitrary codec version still under development.  

I cannot agree more. Freeze a version of Opus, and let's check the quality of the codec. If it passes the quality expectations, it will become a standard. 

-- But before that, clean up the code and the specification and fix the IPR issues. Right now the codec does not pass the "admin" part of requirements.

Kind regards,
/Anisse