Re: [codec] Adopt draft-valin-codec-results as a working group item

Erik Norvell <erik.norvell@ericsson.com> Fri, 09 September 2011 09:25 UTC

Return-Path: <erik.norvell@ericsson.com>
X-Original-To: codec@ietfa.amsl.com
Delivered-To: codec@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DDC2A21F862F for <codec@ietfa.amsl.com>; Fri, 9 Sep 2011 02:25:12 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.404
X-Spam-Level:
X-Spam-Status: No, score=-6.404 tagged_above=-999 required=5 tests=[AWL=0.195, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KRpGBUPByYSf for <codec@ietfa.amsl.com>; Fri, 9 Sep 2011 02:25:12 -0700 (PDT)
Received: from mailgw10.se.ericsson.net (mailgw10.se.ericsson.net [193.180.251.61]) by ietfa.amsl.com (Postfix) with ESMTP id C32E321F85C7 for <codec@ietf.org>; Fri, 9 Sep 2011 02:25:11 -0700 (PDT)
X-AuditID: c1b4fb3d-b7c47ae000000b17-59-4e69dbe600f8
Received: from esessmw0197.eemea.ericsson.se (Unknown_Domain [153.88.253.125]) by mailgw10.se.ericsson.net (Symantec Mail Security) with SMTP id 61.CE.02839.6EBD96E4; Fri, 9 Sep 2011 11:27:02 +0200 (CEST)
Received: from ESESSCMS0351.eemea.ericsson.se ([169.254.2.199]) by esessmw0197.eemea.ericsson.se ([153.88.115.87]) with mapi; Fri, 9 Sep 2011 11:27:01 +0200
From: Erik Norvell <erik.norvell@ericsson.com>
To: Ron <ron@debian.org>, "codec@ietf.org" <codec@ietf.org>
Date: Fri, 9 Sep 2011 11:27:00 +0200
Thread-Topic: [codec] Adopt draft-valin-codec-results as a working group item
Thread-Index: Acxudq7CnX8K5kgTRGWr/WGNE8lDxQAWpRPQ
Message-ID: <027A93CE4A670242BD91A44E37105AEF2EE0FAA404@ESESSCMS0351.eemea.ericsson.se>
References: <4E52C241.2010208@jdrosen.net> <4E67BC0D.9070208@jdrosen.net> <6A58A83F7040374B9FB4EEEDBD835512A3FB7C@LHREML503-MBX.china.huawei.com> <20110908222842.GW17064@audi.shelbyville.oz>
In-Reply-To: <20110908222842.GW17064@audi.shelbyville.oz>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Brightmail-Tracker: AAAAAA==
Subject: Re: [codec] Adopt draft-valin-codec-results as a working group item
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 09 Sep 2011 09:25:13 -0000

Hi all,

I support excluding sections 2 and 3 from the document. Not because they are old, but because the codec has changed since the tests were conducted and the results cannot represent the Opus performance. I agree the document should only contain tests considered meaningful and valid to this WG and removing these sections would increase the quality of the document.

Best regards,
Erik 

> -----Original Message-----
> From: codec-bounces@ietf.org [mailto:codec-bounces@ietf.org] 
> On Behalf Of Ron
> Sent: den 9 september 2011 00:29
> To: codec@ietf.org
> Subject: Re: [codec] Adopt draft-valin-codec-results as a 
> working group item
> 
> On Thu, Sep 08, 2011 at 09:54:27AM +0000, Anisse Taleb wrote:
> > If I recall right, there was a consensus at the meeting in 
> Quebec that 
> > this document will be assigned to Christian as editor and that work 
> > will progress into making this document as good and as complete as 
> > possible. Thus, IMHO, the document is a good starting point for 
> > further editing. When it comes to the contents of the document, I 
> > think what the industry would like to see is a full 
> characterization 
> > of the Opus codec and its features as an internet codec 
> which is a core objective of this work.
> 
> I missed being (tele)present for the Quebec meeting, so I 
> can't comment on what was agreed there, but I agree this 
> document seems like a good starting point for a WG item, that 
> should be added to as further test results become available 
> which are accepted by the group as being meaningful and valid.
> 
> > On the other hand, tests performed prior to the adoption 
> (or freeze) 
> > of the codec do not offer much in terms of characterization 
> since they 
> > relate to a earlier version of the codec and will not accurately 
> > reflect the performance of the codec.
> 
> I don't think they are being misrepresented as such though, do you?
> For CELT at least, the versions of the codec that were tested 
> are still available, and documenting these tests provides 
> useful historical context for people who may wish to repeat 
> similar tests on the final codec and gauge the level of 
> improvement that has been made since this group began its 
> work on them.
> 
> > In summary, section 2) and 3) should be removed
> 
> ... so I disagree with this as a blanket statement if your 
> only reasoning is "this documents history, not the present".
> 
> I think possibly the SILK tests could be clarified to 
> identify exactly which version of SILK was used for them.  
> And if you have reason to doubt the validity of individual 
> tests for reasons other than "they are old", then that should 
> be considered.  But otherwise, I think these sections provide 
> useful historical information of both the improvements that 
> were made and the process of testing that guided them, and 
> that the value and transparency of this document would not be 
> improved by removing them.
> 
> One of the aims of this group was to demonstrate the benefits 
> of using an open and evolutionary methodology to create 
> something better than could be achieved by a "develop behind 
> closed doors, winner takes all shootout" process.  I think 
> documenting the history of testing, and the results that were 
> seen, is a valuable part of describing that process and its 
> results for the people who will come after us, and that 
> simply discarding it now that it has already been compiled 
> would only do them a disservice.
> 
> I don't think this is inconsistent with the desire that you 
> say was expressed in Quebec for it to be "as good and 
> complete as possible".
> Discarding these sections would only make it less complete, 
> without actually making it better in any way.
> 
> But, that said ...
> 
> > and the focus shifted towards a more detailed section 4).
> 
> ... I agree there isn't a lot to be gained by bikeshedding 
> over these old tests now, unless there are serious concerns 
> about the validity of the process they employed, that would 
> make them bad examples to follow, for people who wished to do 
> later testing.
> 
> The focus of future editing should mainly be on the relevance 
> and validity of any new tests that people wish to add to it 
> as they come.
> 
> I completely agree that it's highly desirable for us to avoid 
> misleading "the industry" about the characteristics of the 
> final RFC defined codec, so results should only be added 
> there after WG scrutiny and consensus.
> 
> Cheers,
> Ron
> 
> 
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec
>