Re: [codec] draft test and processing plan for the IETF Codec

["Benjamin M. Schwartz" <[e-mail]>]

On 04/13/2011 03:32 AM, Anisse Taleb wrote:
> Please find attached a first draft of a test plan of the IETF codec (Opus). 

Thank you for drawing up this test plan, which clearly required a great
deal of thought.  The results of such testing would certainly be very
interesting to many.

However, I think the execution of such a test is clearly _not_ an
appropriate prerequisite for publishing a Proposed Standard.  By my
calculations, the draft plan presently calls for over 1300 hours of
listening tests, counting only audio being played, estimating 10-second
samples and the minimum number of listeners.  Even if many listeners are
listening in parallel, and overheads (such as delays between samples) are
low, conducting such a test would still take many months.

Such an extensive, expensive battery of tests can hardly be justified on
some arbitrary codec version still under development.  It can only be
justified if the codec being tested is not going to change, so that the
sponsoring organizations can use the results to determine whether the
codec meets their performance goals.

Let's standardize, and then invite ultra-comprehensive systematic
characterization.

--Ben