Re: [codec] draft test and processing plan for the IETF Codec

Erik Norvell <> Wed, 13 April 2011 14:48 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id E99E1E0796 for <>; Wed, 13 Apr 2011 07:48:48 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -6.599
X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id xrrihAXqOvfk for <>; Wed, 13 Apr 2011 07:48:48 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id 09E23E0793 for <>; Wed, 13 Apr 2011 07:48:47 -0700 (PDT)
X-AuditID: c1b4fb39-b7c6dae0000023f2-59-4da5b7cf5947
Received: from (Unknown_Domain []) by (Symantec Mail Security) with SMTP id 9A.48.09202.FC7B5AD4; Wed, 13 Apr 2011 16:48:47 +0200 (CEST)
Received: from ([]) by ([]) with mapi; Wed, 13 Apr 2011 16:48:47 +0200
From: Erik Norvell <>
To: "" <>, Anisse Taleb <>
Date: Wed, 13 Apr 2011 16:48:45 +0200
Thread-Topic: [codec] draft test and processing plan for the IETF Codec
Thread-Index: Acv54A/vm/J7c2g7RCKYlZpzEk3w1gABwdTg
Message-ID: <>
References: <> <>
In-Reply-To: <>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Brightmail-Tracker: AAAAAA==
Cc: "" <>
Subject: Re: [codec] draft test and processing plan for the IETF Codec
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 13 Apr 2011 14:48:49 -0000

Hi Ben, all

If the codec is not ready for testing, then I cannot see how it could be ready for standardization. To me the steps would be 

- freeze the codec when it is stable
- test and evaluate
- check if requirements are met
	a) if yes standardize
	b) if not do not standarize and rather go back and improve

Informal testing should still be done during development to eliminate the risk of b).

I also think the encumbrance of the codec is unclear at this point and I don't think rushing to finalize the standard would serve the purpose of this WG. Due to the encumbrance there may still be changes required which may affect the quality, and the final testing should begin after this has been resolved.


> -----Original Message-----
> From: [] 
> On Behalf Of Benjamin M. Schwartz
> Sent: den 13 april 2011 15:38
> To: Anisse Taleb
> Cc:
> Subject: Re: [codec] draft test and processing plan for the IETF Codec
> On 04/13/2011 03:32 AM, Anisse Taleb wrote:
> > Please find attached a first draft of a test plan of the 
> IETF codec (Opus). 
> Thank you for drawing up this test plan, which clearly 
> required a great deal of thought.  The results of such 
> testing would certainly be very interesting to many.
> However, I think the execution of such a test is clearly 
> _not_ an appropriate prerequisite for publishing a Proposed 
> Standard.  By my calculations, the draft plan presently calls 
> for over 1300 hours of listening tests, counting only audio 
> being played, estimating 10-second samples and the minimum 
> number of listeners.  Even if many listeners are listening in 
> parallel, and overheads (such as delays between samples) are 
> low, conducting such a test would still take many months.
> Such an extensive, expensive battery of tests can hardly be 
> justified on some arbitrary codec version still under 
> development.  It can only be justified if the codec being 
> tested is not going to change, so that the sponsoring 
> organizations can use the results to determine whether the 
> codec meets their performance goals.
> Let's standardize, and then invite ultra-comprehensive 
> systematic characterization.
> --Ben