Re: [codec] A concrete proposal for requirements and testing
"Benjamin M. Schwartz" <bmschwar@fas.harvard.edu> Wed, 06 April 2011 14:19 UTC
Return-Path: <bmschwar@fas.harvard.edu>
X-Original-To: codec@core3.amsl.com
Delivered-To: codec@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id E7FC128C0F1 for <codec@core3.amsl.com>; Wed, 6 Apr 2011 07:19:30 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.599
X-Spam-Level:
X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FS7icr8A-QJP for <codec@core3.amsl.com>; Wed, 6 Apr 2011 07:19:29 -0700 (PDT)
Received: from us18.unix.fas.harvard.edu (us18.unix.fas.harvard.edu [140.247.35.198]) by core3.amsl.com (Postfix) with ESMTP id B6A9F28C0DB for <codec@ietf.org>; Wed, 6 Apr 2011 07:19:29 -0700 (PDT)
Received: from us18.unix.fas.harvard.edu (localhost.localdomain [127.0.0.1]) by us18.unix.fas.harvard.edu (Postfix) with ESMTP id 1BC1E4D5BEB; Wed, 6 Apr 2011 10:21:13 -0400 (EDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=fas.harvard.edu; h= message-id:date:from:reply-to:mime-version:to:cc:subject :references:in-reply-to:content-type; s=mail; bh=XuNPgO//NW9gbe1 wc6h5II+CQg/fh+lUVlAlnq+pqy8=; b=wbGThk/ioWxfTxiR6iypdHNQ3guVqB+ fRkzMft4SPj/iY+Dw1M9O174BkBP/6EDBSn/QUgxdCIcOF7TIUUMjV28/22tq5b+ sB6/U40BfVe4g5wgPIASZnJVWMHJB6h0nvcGedAXcA9Efwe97hE7rYkl8z7Vzk7I ww6JiahZfWho=
DomainKey-Signature: a=rsa-sha1; c=simple; d=fas.harvard.edu; h= message-id:date:from:reply-to:mime-version:to:cc:subject :references:in-reply-to:content-type; q=dns; s=mail; b=X+vhE1s7Y M+A4t1hK7ZTdDtStG99cBIuMOE+cCTw3hSCW32Izczqoopid96cGZPjJ14uoM4An /RJmtSKzDasCqW7LGe8XxizIYJfCFjY8z+o+T1tMxjX5MEYQWOQvMxCVlxeGMYae RDg8299e9+KhF1htXJrRTNhs9IrYBDloh8=
Received: from [192.168.1.141] (c-71-192-160-188.hsd1.nh.comcast.net [71.192.160.188]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: bmschwar@fas) by us18.unix.fas.harvard.edu (Postfix) with ESMTPSA id 0F4FC4D5A6D; Wed, 6 Apr 2011 10:21:13 -0400 (EDT)
Message-ID: <4D9C76D4.9080205@fas.harvard.edu>
Date: Wed, 06 Apr 2011 10:21:08 -0400
From: "Benjamin M. Schwartz" <bmschwar@fas.harvard.edu>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Lightning/1.0b2 Thunderbird/3.1.8
MIME-Version: 1.0
To: Anisse Taleb <anisse.taleb@huawei.com>
References: <64212FE1AE068044AD567CCB214073F123A10234@MAIL2.octasic.com> <AANLkTinrS6e+i_VE5Q91gbNV5yhwi3Ad+wZOUsKYGNZn@mail.gmail.com> <20110331124329.216219p5zb5npnpc@www.usherbrooke.ca> <AANLkTimZF6=vaOCJ7G8XyRoKx5cSKW4YHmSyDsE=s3EK@mail.gmail.com> <4D94BAF5.40501@fas.harvard.edu> <F5AD4C2E5FBF304ABAE7394E9979AF7C26BC4826@LHREML503-MBX.china.huawei.com>
In-Reply-To: <F5AD4C2E5FBF304ABAE7394E9979AF7C26BC4826@LHREML503-MBX.china.huawei.com>
X-Enigmail-Version: 1.1.2
Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="------------enig96A1A7B55BE48786F3041743"
Cc: "codec@ietf.org" <codec@ietf.org>
Subject: Re: [codec] A concrete proposal for requirements and testing
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
Reply-To: bens@alum.mit.edu
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Apr 2011 14:19:31 -0000
On 04/06/2011 04:42 AM, Anisse Taleb wrote: > So far, only a subset of signal types have been used This will always be true. There are infinitely many potential signal types, and even in broad categories, there are a combinatorial explosion of use cases. Most of them will necessarily remain untested. Testing all, or even most, use cases is _not_ a standardization requirement for this working group. The experts in audio quality testing with whom I have spoken have consistently recommended identifying a small number of core use cases and focusing testing effort on them. That is exactly what we have done. > and verifying the requirements requires a thorough mirrored tests that includes many signal types encountered in the foreseen applications and usecases. I am not familiar with the term "mirrored test", but I presume it refers to performing similar experiments at multiple sites. I don't find this appealing, but if you do then you may volunteer to administer such a site and list the tests you are willing to perform. If those who are calling here for more testing do not volunteer to perform the tests, then we will have no logical choice but to standardize with the testing we have. The tests so far are statistically significant and show Opus performing excellently in a wide range of applications. Because the Opus encoder is non-normative, further pre-standardization testing is highly unlikely to produce results that impact the specification. At this point we can let the market decide, and resolve any remaining deficits by improving the encoder after standardization. --Ben
- [codec] A concrete proposal for requirements and … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Roman Shpount
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Roman Shpount
- Re: [codec] A concrete proposal for requirements … Benjamin M. Schwartz
- Re: [codec] A concrete proposal for requirements … Roni Even
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Jan Skoglund
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- Re: [codec] A concrete proposal for requirements … Erik Norvell
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Benjamin M. Schwartz
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Peter Saint-Andre
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Benjamin M. Schwartz
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Peter Saint-Andre
- Re: [codec] A concrete proposal for requirements … Timothy B. Terriberry
- Re: [codec] A concrete proposal for requirements … Stephan Wenger
- Re: [codec] A concrete proposal for requirements … Monty Montgomery
- Re: [codec] A concrete proposal for requirements … Timothy B. Terriberry
- Re: [codec] A concrete proposal for requirements … Stephan Wenger
- Re: [codec] A concrete proposal for requirements … Gregory Maxwell
- Re: [codec] A concrete proposal for requirements … Monty Montgomery
- Re: [codec] A concrete proposal for requirements … Koen Vos
- Re: [codec] A concrete proposal for requirements … Roman Shpount
- Re: [codec] A concrete proposal for requirements … Koen Vos
- Re: [codec] A concrete proposal for requirements … Stephan Wenger
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Gregory Maxwell
- Re: [codec] A concrete proposal for requirements … Roman Shpount
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Koen Vos
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Roni Even
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Kavan Seggie
- Re: [codec] A concrete proposal for requirements … Roni Even
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Roni Even
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Roni Even
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Stephan Wenger
- Re: [codec] A concrete proposal for requirements … Kat Walsh
- Re: [codec] A concrete proposal for requirements … Stefan Hacker
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Serge Smirnoff
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- [codec] Chairs and consensus Cullen Jennings