Re: [codec] Adopt draft-valin-codec-results as a working group item

Erik Norvell <> Fri, 09 September 2011 09:25 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id DDC2A21F862F for <>; Fri, 9 Sep 2011 02:25:12 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -6.404
X-Spam-Status: No, score=-6.404 tagged_above=-999 required=5 tests=[AWL=0.195, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id KRpGBUPByYSf for <>; Fri, 9 Sep 2011 02:25:12 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id C32E321F85C7 for <>; Fri, 9 Sep 2011 02:25:11 -0700 (PDT)
X-AuditID: c1b4fb3d-b7c47ae000000b17-59-4e69dbe600f8
Received: from (Unknown_Domain []) by (Symantec Mail Security) with SMTP id 61.CE.02839.6EBD96E4; Fri, 9 Sep 2011 11:27:02 +0200 (CEST)
Received: from ([]) by ([]) with mapi; Fri, 9 Sep 2011 11:27:01 +0200
From: Erik Norvell <>
To: Ron <>, "" <>
Date: Fri, 9 Sep 2011 11:27:00 +0200
Thread-Topic: [codec] Adopt draft-valin-codec-results as a working group item
Thread-Index: Acxudq7CnX8K5kgTRGWr/WGNE8lDxQAWpRPQ
Message-ID: <>
References: <> <> <> <20110908222842.GW17064@audi.shelbyville.oz>
In-Reply-To: <20110908222842.GW17064@audi.shelbyville.oz>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Brightmail-Tracker: AAAAAA==
Subject: Re: [codec] Adopt draft-valin-codec-results as a working group item
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 09 Sep 2011 09:25:13 -0000

Hi all,

I support excluding sections 2 and 3 from the document. Not because they are old, but because the codec has changed since the tests were conducted and the results cannot represent the Opus performance. I agree the document should only contain tests considered meaningful and valid to this WG and removing these sections would increase the quality of the document.

Best regards,

> -----Original Message-----
> From: [] 
> On Behalf Of Ron
> Sent: den 9 september 2011 00:29
> To:
> Subject: Re: [codec] Adopt draft-valin-codec-results as a 
> working group item
> On Thu, Sep 08, 2011 at 09:54:27AM +0000, Anisse Taleb wrote:
> > If I recall right, there was a consensus at the meeting in 
> Quebec that 
> > this document will be assigned to Christian as editor and that work 
> > will progress into making this document as good and as complete as 
> > possible. Thus, IMHO, the document is a good starting point for 
> > further editing. When it comes to the contents of the document, I 
> > think what the industry would like to see is a full 
> characterization 
> > of the Opus codec and its features as an internet codec 
> which is a core objective of this work.
> I missed being (tele)present for the Quebec meeting, so I 
> can't comment on what was agreed there, but I agree this 
> document seems like a good starting point for a WG item, that 
> should be added to as further test results become available 
> which are accepted by the group as being meaningful and valid.
> > On the other hand, tests performed prior to the adoption 
> (or freeze) 
> > of the codec do not offer much in terms of characterization 
> since they 
> > relate to a earlier version of the codec and will not accurately 
> > reflect the performance of the codec.
> I don't think they are being misrepresented as such though, do you?
> For CELT at least, the versions of the codec that were tested 
> are still available, and documenting these tests provides 
> useful historical context for people who may wish to repeat 
> similar tests on the final codec and gauge the level of 
> improvement that has been made since this group began its 
> work on them.
> > In summary, section 2) and 3) should be removed
> ... so I disagree with this as a blanket statement if your 
> only reasoning is "this documents history, not the present".
> I think possibly the SILK tests could be clarified to 
> identify exactly which version of SILK was used for them.  
> And if you have reason to doubt the validity of individual 
> tests for reasons other than "they are old", then that should 
> be considered.  But otherwise, I think these sections provide 
> useful historical information of both the improvements that 
> were made and the process of testing that guided them, and 
> that the value and transparency of this document would not be 
> improved by removing them.
> One of the aims of this group was to demonstrate the benefits 
> of using an open and evolutionary methodology to create 
> something better than could be achieved by a "develop behind 
> closed doors, winner takes all shootout" process.  I think 
> documenting the history of testing, and the results that were 
> seen, is a valuable part of describing that process and its 
> results for the people who will come after us, and that 
> simply discarding it now that it has already been compiled 
> would only do them a disservice.
> I don't think this is inconsistent with the desire that you 
> say was expressed in Quebec for it to be "as good and 
> complete as possible".
> Discarding these sections would only make it less complete, 
> without actually making it better in any way.
> But, that said ...
> > and the focus shifted towards a more detailed section 4).
> ... I agree there isn't a lot to be gained by bikeshedding 
> over these old tests now, unless there are serious concerns 
> about the validity of the process they employed, that would 
> make them bad examples to follow, for people who wished to do 
> later testing.
> The focus of future editing should mainly be on the relevance 
> and validity of any new tests that people wish to add to it 
> as they come.
> I completely agree that it's highly desirable for us to avoid 
> misleading "the industry" about the characteristics of the 
> final RFC defined codec, so results should only be added 
> there after WG scrutiny and consensus.
> Cheers,
> Ron
> _______________________________________________
> codec mailing list