Re: [codec] draft test and processing plan for the IETF Codec
"Benjamin M. Schwartz" <bmschwar@fas.harvard.edu> Wed, 13 April 2011 13:38 UTC
Return-Path: <bmschwar@fas.harvard.edu>
X-Original-To: codec@ietfc.amsl.com
Delivered-To: codec@ietfc.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id 80CC1E0690 for <codec@ietfc.amsl.com>; Wed, 13 Apr 2011 06:38:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.599
X-Spam-Level:
X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ib+tt2oygPzV for <codec@ietfc.amsl.com>; Wed, 13 Apr 2011 06:38:20 -0700 (PDT)
Received: from us17.unix.fas.harvard.edu (us17.unix.fas.harvard.edu [140.247.35.197]) by ietfc.amsl.com (Postfix) with ESMTP id A8207E066B for <codec@ietf.org>; Wed, 13 Apr 2011 06:38:20 -0700 (PDT)
Received: from us17.unix.fas.harvard.edu (localhost.localdomain [127.0.0.1]) by us17.unix.fas.harvard.edu (Postfix) with ESMTP id 2AD0541E8DA; Wed, 13 Apr 2011 09:38:19 -0400 (EDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=fas.harvard.edu; h= message-id:date:from:reply-to:mime-version:to:cc:subject :references:in-reply-to:content-type; s=mail; bh=dEEH08sgvU4qzH7 gGcI7LbtwWbVHMvmiJwnQQ73yH+4=; b=T/tS/iw9qOQY1ufpQPnJC9TneuPMkcQ U71fHTUXURC7aX9vOJiLX9CmUc1zJ2pkPYxIeATUzgInadFMXFz463lOSysh3Ymn oqAEah0D6caAw0pYL76ynW+9wVsop7N9VA/j0u/Yz6HTFiq7sJ1tne+m3XVflSAc hmZdPjXxOfHI=
DomainKey-Signature: a=rsa-sha1; c=simple; d=fas.harvard.edu; h= message-id:date:from:reply-to:mime-version:to:cc:subject :references:in-reply-to:content-type; q=dns; s=mail; b=RmDGcQb7Q AqnAp4TKuNGajntm6LJ8TzxILZGJ30lEa0WbNhTjGxL9RbFRADQmdU/CP/Pd7T8d lknbDDjUzghBpnM5Z/JpHPOOFHBiJoeCrxteSypsZQ7LglKCALiQ5N/1HcHp1SYM f8wx/1LAYBgK+9c4mzCk1phsLMVCbsLUso=
Received: from [192.168.1.141] (c-71-192-160-188.hsd1.nh.comcast.net [71.192.160.188]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: bmschwar@fas) by us17.unix.fas.harvard.edu (Postfix) with ESMTPSA id C291941E8D7; Wed, 13 Apr 2011 09:38:18 -0400 (EDT)
Message-ID: <4DA5A748.2050401@fas.harvard.edu>
Date: Wed, 13 Apr 2011 09:38:16 -0400
From: "Benjamin M. Schwartz" <bmschwar@fas.harvard.edu>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Lightning/1.0b2 Thunderbird/3.1.8
MIME-Version: 1.0
To: Anisse Taleb <anisse.taleb@huawei.com>
References: <F5AD4C2E5FBF304ABAE7394E9979AF7C26BC684E@LHREML503-MBX.china.huawei.com>
In-Reply-To: <F5AD4C2E5FBF304ABAE7394E9979AF7C26BC684E@LHREML503-MBX.china.huawei.com>
X-Enigmail-Version: 1.1.2
Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="------------enig8814F0833D05EB7DA603513D"
Cc: "codec@ietf.org" <codec@ietf.org>
Subject: Re: [codec] draft test and processing plan for the IETF Codec
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
Reply-To: bens@alum.mit.edu
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 13 Apr 2011 13:38:21 -0000
On 04/13/2011 03:32 AM, Anisse Taleb wrote: > Please find attached a first draft of a test plan of the IETF codec (Opus). Thank you for drawing up this test plan, which clearly required a great deal of thought. The results of such testing would certainly be very interesting to many. However, I think the execution of such a test is clearly _not_ an appropriate prerequisite for publishing a Proposed Standard. By my calculations, the draft plan presently calls for over 1300 hours of listening tests, counting only audio being played, estimating 10-second samples and the minimum number of listeners. Even if many listeners are listening in parallel, and overheads (such as delays between samples) are low, conducting such a test would still take many months. Such an extensive, expensive battery of tests can hardly be justified on some arbitrary codec version still under development. It can only be justified if the codec being tested is not going to change, so that the sponsoring organizations can use the results to determine whether the codec meets their performance goals. Let's standardize, and then invite ultra-comprehensive systematic characterization. --Ben
- [codec] draft test and processing plan for the IE… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Schulz, Edward D (Ed)
- Re: [codec] draft test and processing plan for th… Jean-Marc Valin
- Re: [codec] draft test and processing plan for th… Benjamin M. Schwartz
- Re: [codec] draft test and processing plan for th… Stephen Botzko
- Re: [codec] draft test and processing plan for th… Roman Shpount
- Re: [codec] draft test and processing plan for th… Erik Norvell
- Re: [codec] draft test and processing plan for th… Paul Coverdale
- Re: [codec] draft test and processing plan for th… Peter Saint-Andre
- Re: [codec] draft test and processing plan for th… Stephan Wenger
- Re: [codec] draft test and processing plan for th… Benjamin M. Schwartz
- Re: [codec] draft test and processing plan for th… Stephen Botzko
- Re: [codec] draft test and processing plan for th… Peter Saint-Andre
- Re: [codec] draft test and processing plan for th… Stephen Botzko
- Re: [codec] draft test and processing plan for th… Ron
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Jean-Marc Valin
- Re: [codec] draft test and processing plan for th… Gregory Maxwell
- Re: [codec] draft test and processing plan for th… Koen Vos
- Re: [codec] draft test and processing plan for th… Paul Coverdale
- Re: [codec] draft test and processing plan for th… Jean-Marc Valin
- Re: [codec] draft test and processing plan for th… Cullen Jennings
- Re: [codec] draft test and processing plan for th… Jean-Marc Valin
- Re: [codec] draft test and processing plan for th… Jean-Marc Valin
- Re: [codec] draft test and processing plan for th… Koen Vos
- Re: [codec] draft test and processing plan for th… Jean-Marc Valin
- Re: [codec] draft test and processing plan for th… Cullen Jennings
- Re: [codec] draft test and processing plan for th… Roman Shpount
- Re: [codec] draft test and processing plan for th… Koen Vos
- Re: [codec] draft test and processing plan for th… Paul Coverdale
- Re: [codec] draft test and processing plan for th… Jean-Marc Valin
- Re: [codec] draft test and processing plan for th… Koen Vos
- Re: [codec] draft test and processing plan for th… Paul Coverdale
- Re: [codec] draft test and processing plan for th… Koen Vos
- Re: [codec] draft test and processing plan for th… Paul Coverdale
- Re: [codec] draft test and processing plan for th… Koen Vos
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Stephen Botzko
- Re: [codec] draft test and processing plan for th… Ron
- Re: [codec] draft test and processing plan for th… Ron
- Re: [codec] draft test and processing plan for th… Stephen Botzko
- Re: [codec] draft test and processing plan for th… Stephen Botzko
- Re: [codec] draft test and processing plan for th… David Virette
- Re: [codec] draft test and processing plan for th… Ron
- Re: [codec] draft test and processing plan for th… Koen Vos
- Re: [codec] draft test and processing plan for th… Jean-Marc Valin
- Re: [codec] draft test and processing plan for th… Stephen Botzko
- Re: [codec] draft test and processing plan for th… Jean-Marc Valin
- Re: [codec] draft test and processing plan for th… Roman Shpount
- Re: [codec] draft test and processing plan for th… Koen Vos
- Re: [codec] draft test and processing plan for th… Michael Ramalho (mramalho)
- Re: [codec] draft test and processing plan for th… Roman Shpount
- Re: [codec] draft test and processing plan for th… David Virette
- Re: [codec] draft test and processing plan for th… David Virette
- Re: [codec] draft test and processing plan for th… David Virette
- Re: [codec] draft test and processing plan for th… Jean-Marc Valin
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Koen Vos
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Jean-Marc Valin
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Paul Coverdale
- Re: [codec] draft test and processing plan for th… Jean-Marc Valin
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Ron
- Re: [codec] draft test and processing plan for th… Monty Montgomery
- Re: [codec] draft test and processing plan for th… Koen Vos
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Cullen Jennings
- Re: [codec] draft test and processing plan for th… Koen Vos
- Re: [codec] draft test and processing plan for th… Koen Vos
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Anisse Taleb
- Re: [codec] draft test and processing plan for th… Christian Hoene
- Re: [codec] draft test and processing plan for th… Christian Hoene