Re: [codec] draft test and processing plan for the IETF Codec

[Erik Norvell <[e-mail]>]

Hi Ben, all

If the codec is not ready for testing, then I cannot see how it could be ready for standardization. To me the steps would be 

- freeze the codec when it is stable
- test and evaluate
- check if requirements are met
	a) if yes standardize
	b) if not do not standarize and rather go back and improve

Informal testing should still be done during development to eliminate the risk of b).

I also think the encumbrance of the codec is unclear at this point and I don't think rushing to finalize the standard would serve the purpose of this WG. Due to the encumbrance there may still be changes required which may affect the quality, and the final testing should begin after this has been resolved.

Best,
Erik 


> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] 
> On Behalf Of Benjamin M. Schwartz
> Sent: den 13 april 2011 15:38
> To: Anisse Taleb
> Cc: [e-mail]
> Subject: Re: [codec] draft test and processing plan for the IETF Codec
> 
> On 04/13/2011 03:32 AM, Anisse Taleb wrote:
> > Please find attached a first draft of a test plan of the 
> IETF codec (Opus). 
> 
> Thank you for drawing up this test plan, which clearly 
> required a great deal of thought.  The results of such 
> testing would certainly be very interesting to many.
> 
> However, I think the execution of such a test is clearly 
> _not_ an appropriate prerequisite for publishing a Proposed 
> Standard.  By my calculations, the draft plan presently calls 
> for over 1300 hours of listening tests, counting only audio 
> being played, estimating 10-second samples and the minimum 
> number of listeners.  Even if many listeners are listening in 
> parallel, and overheads (such as delays between samples) are 
> low, conducting such a test would still take many months.
> 
> Such an extensive, expensive battery of tests can hardly be 
> justified on some arbitrary codec version still under 
> development.  It can only be justified if the codec being 
> tested is not going to change, so that the sponsoring 
> organizations can use the results to determine whether the 
> codec meets their performance goals.
> 
> Let's standardize, and then invite ultra-comprehensive 
> systematic characterization.
> 
> --Ben
> 
>