Re: [codec] A concrete proposal for requirements and testing

"Benjamin M. Schwartz" <bmschwar@fas.harvard.edu> Wed, 06 April 2011 14:19 UTC

Return-Path: <bmschwar@fas.harvard.edu>
X-Original-To: codec@core3.amsl.com
Delivered-To: codec@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id E7FC128C0F1 for <codec@core3.amsl.com>; Wed, 6 Apr 2011 07:19:30 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.599
X-Spam-Level:
X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FS7icr8A-QJP for <codec@core3.amsl.com>; Wed, 6 Apr 2011 07:19:29 -0700 (PDT)
Received: from us18.unix.fas.harvard.edu (us18.unix.fas.harvard.edu [140.247.35.198]) by core3.amsl.com (Postfix) with ESMTP id B6A9F28C0DB for <codec@ietf.org>; Wed, 6 Apr 2011 07:19:29 -0700 (PDT)
Received: from us18.unix.fas.harvard.edu (localhost.localdomain [127.0.0.1]) by us18.unix.fas.harvard.edu (Postfix) with ESMTP id 1BC1E4D5BEB; Wed, 6 Apr 2011 10:21:13 -0400 (EDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=fas.harvard.edu; h= message-id:date:from:reply-to:mime-version:to:cc:subject :references:in-reply-to:content-type; s=mail; bh=XuNPgO//NW9gbe1 wc6h5II+CQg/fh+lUVlAlnq+pqy8=; b=wbGThk/ioWxfTxiR6iypdHNQ3guVqB+ fRkzMft4SPj/iY+Dw1M9O174BkBP/6EDBSn/QUgxdCIcOF7TIUUMjV28/22tq5b+ sB6/U40BfVe4g5wgPIASZnJVWMHJB6h0nvcGedAXcA9Efwe97hE7rYkl8z7Vzk7I ww6JiahZfWho=
DomainKey-Signature: a=rsa-sha1; c=simple; d=fas.harvard.edu; h= message-id:date:from:reply-to:mime-version:to:cc:subject :references:in-reply-to:content-type; q=dns; s=mail; b=X+vhE1s7Y M+A4t1hK7ZTdDtStG99cBIuMOE+cCTw3hSCW32Izczqoopid96cGZPjJ14uoM4An /RJmtSKzDasCqW7LGe8XxizIYJfCFjY8z+o+T1tMxjX5MEYQWOQvMxCVlxeGMYae RDg8299e9+KhF1htXJrRTNhs9IrYBDloh8=
Received: from [192.168.1.141] (c-71-192-160-188.hsd1.nh.comcast.net [71.192.160.188]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: bmschwar@fas) by us18.unix.fas.harvard.edu (Postfix) with ESMTPSA id 0F4FC4D5A6D; Wed, 6 Apr 2011 10:21:13 -0400 (EDT)
Message-ID: <4D9C76D4.9080205@fas.harvard.edu>
Date: Wed, 06 Apr 2011 10:21:08 -0400
From: "Benjamin M. Schwartz" <bmschwar@fas.harvard.edu>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Lightning/1.0b2 Thunderbird/3.1.8
MIME-Version: 1.0
To: Anisse Taleb <anisse.taleb@huawei.com>
References: <64212FE1AE068044AD567CCB214073F123A10234@MAIL2.octasic.com> <AANLkTinrS6e+i_VE5Q91gbNV5yhwi3Ad+wZOUsKYGNZn@mail.gmail.com> <20110331124329.216219p5zb5npnpc@www.usherbrooke.ca> <AANLkTimZF6=vaOCJ7G8XyRoKx5cSKW4YHmSyDsE=s3EK@mail.gmail.com> <4D94BAF5.40501@fas.harvard.edu> <F5AD4C2E5FBF304ABAE7394E9979AF7C26BC4826@LHREML503-MBX.china.huawei.com>
In-Reply-To: <F5AD4C2E5FBF304ABAE7394E9979AF7C26BC4826@LHREML503-MBX.china.huawei.com>
X-Enigmail-Version: 1.1.2
Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="------------enig96A1A7B55BE48786F3041743"
Cc: "codec@ietf.org" <codec@ietf.org>
Subject: Re: [codec] A concrete proposal for requirements and testing
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
Reply-To: bens@alum.mit.edu
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Apr 2011 14:19:31 -0000

On 04/06/2011 04:42 AM, Anisse Taleb wrote:
> So far, only a subset of signal types have been used

This will always be true.  There are infinitely many potential signal
types, and even in broad categories, there are a combinatorial explosion
of use cases.  Most of them will necessarily remain untested.  Testing
all, or even most, use cases is _not_ a standardization requirement for
this working group.

The experts in audio quality testing with whom I have spoken have
consistently recommended identifying a small number of core use cases and
focusing testing effort on them.  That is exactly what we have done.

> and verifying the requirements requires a thorough  mirrored tests that includes many signal types encountered in the foreseen applications and usecases.

I am not familiar with the term "mirrored test", but I presume it refers
to performing similar experiments at multiple sites.  I don't find this
appealing, but if you do then you may volunteer to administer such a site
and list the tests you are willing to perform.

If those who are calling here for more testing do not volunteer to perform
the tests, then we will have no logical choice but to standardize with the
testing we have.  The tests so far are statistically significant and show
Opus performing excellently in a wide range of applications.

Because the Opus encoder is non-normative, further pre-standardization
testing is highly unlikely to produce results that impact the
specification.  At this point we can let the market decide, and resolve
any remaining deficits by improving the encoder after standardization.

--Ben