Re: [codec] A concrete proposal for requirements and testing
Ron <ron@debian.org> Thu, 07 April 2011 16:46 UTC
Return-Path: <ron@debian.org>
X-Original-To: codec@core3.amsl.com
Delivered-To: codec@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 8F4713A6A20 for <codec@core3.amsl.com>; Thu, 7 Apr 2011 09:46:37 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8ArlFQDsVkTD for <codec@core3.amsl.com>; Thu, 7 Apr 2011 09:46:36 -0700 (PDT)
Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by core3.amsl.com (Postfix) with ESMTP id 7455D3A6A1E for <codec@ietf.org>; Thu, 7 Apr 2011 09:46:35 -0700 (PDT)
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AvsEADjqnU120jym/2dsb2JhbACmB3iIebgShW0EhU6Hdw
Received: from ppp118-210-60-166.lns20.adl2.internode.on.net (HELO audi.shelbyville.oz) ([118.210.60.166]) by ipmail06.adl2.internode.on.net with ESMTP; 08 Apr 2011 02:18:18 +0930
Received: from localhost (localhost [127.0.0.1]) by audi.shelbyville.oz (Postfix) with ESMTP id 0F7D74F8F3; Fri, 8 Apr 2011 02:18:18 +0930 (CST)
X-Virus-Scanned: Debian amavisd-new at audi.shelbyville.oz
Received: from audi.shelbyville.oz ([127.0.0.1]) by localhost (audi.shelbyville.oz [127.0.0.1]) (amavisd-new, port 10024) with LMTP id jQerXiSox8Nt; Fri, 8 Apr 2011 02:18:17 +0930 (CST)
Received: by audi.shelbyville.oz (Postfix, from userid 1000) id 6D7394F8FE; Fri, 8 Apr 2011 02:18:17 +0930 (CST)
Date: Fri, 08 Apr 2011 02:18:17 +0930
From: Ron <ron@debian.org>
To: Stephen Botzko <stephen.botzko@gmail.com>
Message-ID: <20110407164817.GB30415@audi.shelbyville.oz>
References: <64212FE1AE068044AD567CCB214073F123A10234@MAIL2.octasic.com> <F5AD4C2E5FBF304ABAE7394E9979AF7C26BC47FA@LHREML503-MBX.china.huawei.com> <027A93CE4A670242BD91A44E37105AEF17ACA33C36@ESESSCMS0351.eemea.ericsson.se> <20110407125345.GA30415@audi.shelbyville.oz> <BANLkTimeDEPY8va6_MQVztn3YGyTZ2LmVw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <BANLkTimeDEPY8va6_MQVztn3YGyTZ2LmVw@mail.gmail.com>
User-Agent: Mutt/1.5.20 (2009-06-14)
Cc: codec@ietf.org
Subject: Re: [codec] A concrete proposal for requirements and testing
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Apr 2011 16:46:37 -0000
Hi Stephen, On Thu, Apr 07, 2011 at 11:46:05AM -0400, Stephen Botzko wrote: > Hi Ron > > This is not a debugging task, it is a codec characterization/quality > assessment that will be done on the finished product. Yes, I quite understand the formal, dare I say bureaucratic, process that is being proposed here. And whatever you call it, I do wholly respect the scientific method that underpins it (and that was used to create and tune this codec for that matter :). But I like reality too. And the reality is: Have you listened to Opus? Is there actually any doubt in your mind that this is not a wholly adequate (to understate the case) codec for the purpose proposed, with a few minor nits still to be worked out? Since we don't actually have another candidate proposed for the WG to endorse, what exactly would we be shooting off against? The quality of the codec seems among the least of our worries on the present facts. We only have to be better than iLBC to be better than the IETF's current best of breed, and nobody has suggested setting the bar that low. Which isn't to say that I'm against further quality testing. But I think we have enough data to comfortably say it's Good Enough. Better Than Most even. I don't think this should be a blocker for the group moving forward to assess its other aims and publish the decoder spec. Quality improvements can still be made in the encoder after that. > In my view, the proper way to go about it is to first get consensus on the > tests that need to be run, what results are needed, and how the tests need > to be conducted in order to meaningfully compare the results. Then get > folks signed up to actually do the tests. I think you've just described a near impossible task. This group is far too disparate to share a tractable common set of requirements. Interested parties should test their own use cases - and in the event of trouble, report on those tests with the scientifically reproducible detail that you describe. So others can analyse them, and devise solutions. We can equally build consensus on which of those reports are significant and which are 'outliers' to the group charter. > This is not about collaboration vs competition, rather it is about running > distributed tests with results that can integrated and scientifically > analyzed/compared. And recording the test methods, in order to allow other > people in the future to re-do the tests and duplicate the results. Well I'd call that collaboration, but I'm not going to compete over what to call it. My point is, doing that stuff is great. And doing even more of it is greater. But I don't think it needs to hold us up from moving on given the results we already have. What test would Opus have to not pass for you to think it is unsuitable? Best, Ron
- [codec] A concrete proposal for requirements and … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Roman Shpount
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Roman Shpount
- Re: [codec] A concrete proposal for requirements … Benjamin M. Schwartz
- Re: [codec] A concrete proposal for requirements … Roni Even
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Jan Skoglund
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- Re: [codec] A concrete proposal for requirements … Erik Norvell
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Benjamin M. Schwartz
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Peter Saint-Andre
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Benjamin M. Schwartz
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Peter Saint-Andre
- Re: [codec] A concrete proposal for requirements … Timothy B. Terriberry
- Re: [codec] A concrete proposal for requirements … Stephan Wenger
- Re: [codec] A concrete proposal for requirements … Monty Montgomery
- Re: [codec] A concrete proposal for requirements … Timothy B. Terriberry
- Re: [codec] A concrete proposal for requirements … Stephan Wenger
- Re: [codec] A concrete proposal for requirements … Gregory Maxwell
- Re: [codec] A concrete proposal for requirements … Monty Montgomery
- Re: [codec] A concrete proposal for requirements … Koen Vos
- Re: [codec] A concrete proposal for requirements … Roman Shpount
- Re: [codec] A concrete proposal for requirements … Koen Vos
- Re: [codec] A concrete proposal for requirements … Stephan Wenger
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Gregory Maxwell
- Re: [codec] A concrete proposal for requirements … Roman Shpount
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Koen Vos
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Roni Even
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Kavan Seggie
- Re: [codec] A concrete proposal for requirements … Roni Even
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Roni Even
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Roni Even
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Stephan Wenger
- Re: [codec] A concrete proposal for requirements … Kat Walsh
- Re: [codec] A concrete proposal for requirements … Stefan Hacker
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Paul Coverdale
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Serge Smirnoff
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Jean-Marc Valin
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Ron
- Re: [codec] A concrete proposal for requirements … Stephen Botzko
- Re: [codec] A concrete proposal for requirements … Anisse Taleb
- [codec] Chairs and consensus Cullen Jennings