Re: [codec] A concrete proposal for requirements and testing

Ron <ron@debian.org> Thu, 07 April 2011 16:46 UTC

Return-Path: <ron@debian.org>
X-Original-To: codec@core3.amsl.com
Delivered-To: codec@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 8F4713A6A20 for <codec@core3.amsl.com>; Thu, 7 Apr 2011 09:46:37 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8ArlFQDsVkTD for <codec@core3.amsl.com>; Thu, 7 Apr 2011 09:46:36 -0700 (PDT)
Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by core3.amsl.com (Postfix) with ESMTP id 7455D3A6A1E for <codec@ietf.org>; Thu, 7 Apr 2011 09:46:35 -0700 (PDT)
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AvsEADjqnU120jym/2dsb2JhbACmB3iIebgShW0EhU6Hdw
Received: from ppp118-210-60-166.lns20.adl2.internode.on.net (HELO audi.shelbyville.oz) ([118.210.60.166]) by ipmail06.adl2.internode.on.net with ESMTP; 08 Apr 2011 02:18:18 +0930
Received: from localhost (localhost [127.0.0.1]) by audi.shelbyville.oz (Postfix) with ESMTP id 0F7D74F8F3; Fri, 8 Apr 2011 02:18:18 +0930 (CST)
X-Virus-Scanned: Debian amavisd-new at audi.shelbyville.oz
Received: from audi.shelbyville.oz ([127.0.0.1]) by localhost (audi.shelbyville.oz [127.0.0.1]) (amavisd-new, port 10024) with LMTP id jQerXiSox8Nt; Fri, 8 Apr 2011 02:18:17 +0930 (CST)
Received: by audi.shelbyville.oz (Postfix, from userid 1000) id 6D7394F8FE; Fri, 8 Apr 2011 02:18:17 +0930 (CST)
Date: Fri, 08 Apr 2011 02:18:17 +0930
From: Ron <ron@debian.org>
To: Stephen Botzko <stephen.botzko@gmail.com>
Message-ID: <20110407164817.GB30415@audi.shelbyville.oz>
References: <64212FE1AE068044AD567CCB214073F123A10234@MAIL2.octasic.com> <F5AD4C2E5FBF304ABAE7394E9979AF7C26BC47FA@LHREML503-MBX.china.huawei.com> <027A93CE4A670242BD91A44E37105AEF17ACA33C36@ESESSCMS0351.eemea.ericsson.se> <20110407125345.GA30415@audi.shelbyville.oz> <BANLkTimeDEPY8va6_MQVztn3YGyTZ2LmVw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <BANLkTimeDEPY8va6_MQVztn3YGyTZ2LmVw@mail.gmail.com>
User-Agent: Mutt/1.5.20 (2009-06-14)
Cc: codec@ietf.org
Subject: Re: [codec] A concrete proposal for requirements and testing
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Apr 2011 16:46:37 -0000

Hi Stephen,

On Thu, Apr 07, 2011 at 11:46:05AM -0400, Stephen Botzko wrote:
> Hi Ron
> 
> This is not a debugging task, it is a codec characterization/quality
> assessment that will be done on the finished product.

Yes, I quite understand the formal, dare I say bureaucratic, process that
is being proposed here.  And whatever you call it, I do wholly respect the
scientific method that underpins it (and that was used to create and tune
this codec for that matter :).

But I like reality too.  And the reality is:  Have you listened to Opus?

Is there actually any doubt in your mind that this is not a wholly adequate
(to understate the case) codec for the purpose proposed, with a few minor
nits still to be worked out?

Since we don't actually have another candidate proposed for the WG to
endorse, what exactly would we be shooting off against?  The quality of
the codec seems among the least of our worries on the present facts.
We only have to be better than iLBC to be better than the IETF's current
best of breed, and nobody has suggested setting the bar that low.

Which isn't to say that I'm against further quality testing.  But I think
we have enough data to comfortably say it's Good Enough.  Better Than Most
even.  I don't think this should be a blocker for the group moving forward
to assess its other aims and publish the decoder spec.

Quality improvements can still be made in the encoder after that.

> In my view, the proper way to go about it is to first get consensus on the
> tests that need to be run, what results are needed, and how the tests need
> to be conducted in order to meaningfully compare the results.  Then get
> folks signed up to actually do the tests.

I think you've just described a near impossible task.  This group is far
too disparate to share a tractable common set of requirements.
Interested parties should test their own use cases - and in the event of
trouble, report on those tests with the scientifically reproducible detail
that you describe.  So others can analyse them, and devise solutions.

We can equally build consensus on which of those reports are significant
and which are 'outliers' to the group charter.

> This is not about collaboration vs competition, rather it is about running
> distributed tests with results that can integrated and scientifically
> analyzed/compared.  And recording the test methods, in order to allow other
> people in the future to re-do the tests and duplicate the results.

Well I'd call that collaboration, but I'm not going to compete over what
to call it.  My point is, doing that stuff is great.  And doing even more
of it is greater.  But I don't think it needs to hold us up from moving on
given the results we already have.

What test would Opus have to not pass for you to think it is unsuitable?

Best,
Ron