Re: [codec] draft test and processing plan for the IETF Codec

"Benjamin M. Schwartz" <> Wed, 13 April 2011 13:38 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 80CC1E0690 for <>; Wed, 13 Apr 2011 06:38:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -3.599
X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id ib+tt2oygPzV for <>; Wed, 13 Apr 2011 06:38:20 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id A8207E066B for <>; Wed, 13 Apr 2011 06:38:20 -0700 (PDT)
Received: from (localhost.localdomain []) by (Postfix) with ESMTP id 2AD0541E8DA; Wed, 13 Apr 2011 09:38:19 -0400 (EDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed;; h= message-id:date:from:reply-to:mime-version:to:cc:subject :references:in-reply-to:content-type; s=mail; bh=dEEH08sgvU4qzH7 gGcI7LbtwWbVHMvmiJwnQQ73yH+4=; b=T/tS/iw9qOQY1ufpQPnJC9TneuPMkcQ U71fHTUXURC7aX9vOJiLX9CmUc1zJ2pkPYxIeATUzgInadFMXFz463lOSysh3Ymn oqAEah0D6caAw0pYL76ynW+9wVsop7N9VA/j0u/Yz6HTFiq7sJ1tne+m3XVflSAc hmZdPjXxOfHI=
DomainKey-Signature: a=rsa-sha1; c=simple;; h= message-id:date:from:reply-to:mime-version:to:cc:subject :references:in-reply-to:content-type; q=dns; s=mail; b=RmDGcQb7Q AqnAp4TKuNGajntm6LJ8TzxILZGJ30lEa0WbNhTjGxL9RbFRADQmdU/CP/Pd7T8d lknbDDjUzghBpnM5Z/JpHPOOFHBiJoeCrxteSypsZQ7LglKCALiQ5N/1HcHp1SYM f8wx/1LAYBgK+9c4mzCk1phsLMVCbsLUso=
Received: from [] ( []) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: bmschwar@fas) by (Postfix) with ESMTPSA id C291941E8D7; Wed, 13 Apr 2011 09:38:18 -0400 (EDT)
Message-ID: <>
Date: Wed, 13 Apr 2011 09:38:16 -0400
From: "Benjamin M. Schwartz" <>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv: Gecko/20110223 Lightning/1.0b2 Thunderbird/3.1.8
MIME-Version: 1.0
To: Anisse Taleb <>
References: <>
In-Reply-To: <>
X-Enigmail-Version: 1.1.2
Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="------------enig8814F0833D05EB7DA603513D"
Cc: "" <>
Subject: Re: [codec] draft test and processing plan for the IETF Codec
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 13 Apr 2011 13:38:21 -0000

On 04/13/2011 03:32 AM, Anisse Taleb wrote:
> Please find attached a first draft of a test plan of the IETF codec (Opus). 

Thank you for drawing up this test plan, which clearly required a great
deal of thought.  The results of such testing would certainly be very
interesting to many.

However, I think the execution of such a test is clearly _not_ an
appropriate prerequisite for publishing a Proposed Standard.  By my
calculations, the draft plan presently calls for over 1300 hours of
listening tests, counting only audio being played, estimating 10-second
samples and the minimum number of listeners.  Even if many listeners are
listening in parallel, and overheads (such as delays between samples) are
low, conducting such a test would still take many months.

Such an extensive, expensive battery of tests can hardly be justified on
some arbitrary codec version still under development.  It can only be
justified if the codec being tested is not going to change, so that the
sponsoring organizations can use the results to determine whether the
codec meets their performance goals.

Let's standardize, and then invite ultra-comprehensive systematic