[codec] comparitive quality testing

Gregory Maxwell <gmaxwell@juniper.net> Thu, 14 April 2011 14:28 UTC

Return-Path: <gmaxwell@juniper.net>
X-Original-To: codec@ietfc.amsl.com
Delivered-To: codec@ietfc.amsl.com
Received: from localhost (localhost []) by ietfc.amsl.com (Postfix) with ESMTP id 5F14DE08A6 for <codec@ietfc.amsl.com>; Thu, 14 Apr 2011 07:28:17 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.599
X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([]) by localhost (ietfc.amsl.com []) (amavisd-new, port 10024) with ESMTP id LTe4IHTBuRUj for <codec@ietfc.amsl.com>; Thu, 14 Apr 2011 07:28:15 -0700 (PDT)
Received: from exprod7og126.obsmtp.com (exprod7og126.obsmtp.com []) by ietfc.amsl.com (Postfix) with ESMTP id 271C0E08A5 for <codec@ietf.org>; Thu, 14 Apr 2011 07:28:15 -0700 (PDT)
Received: from P-EMHUB01-HQ.jnpr.net ([]) (using TLSv1) by exprod7ob126.postini.com ([]) with SMTP ID DSNKTacEfQe4QhL7lBb+/wxcafBjPbSwOTeg@postini.com; Thu, 14 Apr 2011 07:28:15 PDT
Received: from EMBX01-HQ.jnpr.net ([fe80::c821:7c81:f21f:8bc7]) by P-EMHUB01-HQ.jnpr.net ([fe80::fc92:eb1:759:2c72%11]) with mapi; Thu, 14 Apr 2011 07:25:28 -0700
From: Gregory Maxwell <gmaxwell@juniper.net>
To: Roni Even <ron.even.tlv@gmail.com>, "codec@ietf.org" <codec@ietf.org>
Date: Thu, 14 Apr 2011 07:25:27 -0700
Thread-Topic: comparitive quality testing
Thread-Index: AQHL+q7ZhtE2QkE93ES+tYUuWp4R5A==
Message-ID: <BCB3F026FAC4C145A4A3330806FEFDA93BA8B64643@EMBX01-HQ.jnpr.net>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Subject: [codec] comparitive quality testing
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 14:28:17 -0000

Roni Even [ron.even.tlv@gmail.com] wrote:
> I do not mind if the WG will decide to remove the quality claim and continue
> with developing  a royalty free codec with "good enough" quality not saying
> it is better than other codecs.
> I just think that it should be clear from the charter and requirements what
> is the purpose of the work.

It's funny how we can argue and argue, only to later realize that it comes
down to a simple mutual misunderstanding.

I thought everyone was already on the same page with respect to the 
goals: it's good to be as good as possible, but the chartered purpose
of the WG was only to do a "good quality" codec that was suited
to the listed applications and deployments.

As a developer I know that quality testing is important, and of course
we've done a lot of it of various types.  I strongly believe in scientific
testing, so of course my first instinct would have been to do it here,
but perhaps the reality of the consensus process makes that less
reasonable—as others have pointed out, most other WGs don't really
do anything comparable to quality testing.

Likewise, making sure the outcome is as legally unencumbered as I can
is also very important to me,  but because of the vulgarities of the
process and the law, this isn't something that the working group itself
makes promises about.

So, perhaps it makes sense for the working group to not make any quality
promises in the same way it makes no promises about patents.

It seems clear enough to me now that we can much more easily come to
consensus about achieving good-enough status than about formal testing
gates and requirements.

We should accept your suggestion—drop all the comparative quality
requirements from the requirements draft, and stop discussing comparative
quality here—and then make some progress on technology, rather than
continue bickering about details where we are not going to come to

The market can figure out the comparative quality question on its own.