Re: [codec] Adopt draft-valin-codec-results as a working group item

Hi all,

I support excluding sections 2 and 3 from the document. Not because they are old, but because the codec has changed since the tests were conducted and the results cannot represent the Opus performance. I agree the document should only contain tests considered meaningful and valid to this WG and removing these sections would increase the quality of the document.

Best regards,
Erik 

> -----Original Message-----
> From: codec-bounces@ietf.org [mailto:codec-bounces@ietf.org] 
> On Behalf Of Ron
> Sent: den 9 september 2011 00:29
> To: codec@ietf.org
> Subject: Re: [codec] Adopt draft-valin-codec-results as a 
> working group item
> 
> On Thu, Sep 08, 2011 at 09:54:27AM +0000, Anisse Taleb wrote:
> > If I recall right, there was a consensus at the meeting in 
> Quebec that 
> > this document will be assigned to Christian as editor and that work 
> > will progress into making this document as good and as complete as 
> > possible. Thus, IMHO, the document is a good starting point for 
> > further editing. When it comes to the contents of the document, I 
> > think what the industry would like to see is a full 
> characterization 
> > of the Opus codec and its features as an internet codec 
> which is a core objective of this work.
> 
> I missed being (tele)present for the Quebec meeting, so I 
> can't comment on what was agreed there, but I agree this 
> document seems like a good starting point for a WG item, that 
> should be added to as further test results become available 
> which are accepted by the group as being meaningful and valid.
> 
> > On the other hand, tests performed prior to the adoption 
> (or freeze) 
> > of the codec do not offer much in terms of characterization 
> since they 
> > relate to a earlier version of the codec and will not accurately 
> > reflect the performance of the codec.
> 
> I don't think they are being misrepresented as such though, do you?
> For CELT at least, the versions of the codec that were tested 
> are still available, and documenting these tests provides 
> useful historical context for people who may wish to repeat 
> similar tests on the final codec and gauge the level of 
> improvement that has been made since this group began its 
> work on them.
> 
> > In summary, section 2) and 3) should be removed
> 
> ... so I disagree with this as a blanket statement if your 
> only reasoning is "this documents history, not the present".
> 
> I think possibly the SILK tests could be clarified to 
> identify exactly which version of SILK was used for them.  
> And if you have reason to doubt the validity of individual 
> tests for reasons other than "they are old", then that should 
> be considered.  But otherwise, I think these sections provide 
> useful historical information of both the improvements that 
> were made and the process of testing that guided them, and 
> that the value and transparency of this document would not be 
> improved by removing them.
> 
> One of the aims of this group was to demonstrate the benefits 
> of using an open and evolutionary methodology to create 
> something better than could be achieved by a "develop behind 
> closed doors, winner takes all shootout" process.  I think 
> documenting the history of testing, and the results that were 
> seen, is a valuable part of describing that process and its 
> results for the people who will come after us, and that 
> simply discarding it now that it has already been compiled 
> would only do them a disservice.
> 
> I don't think this is inconsistent with the desire that you 
> say was expressed in Quebec for it to be "as good and 
> complete as possible".
> Discarding these sections would only make it less complete, 
> without actually making it better in any way.
> 
> But, that said ...
> 
> > and the focus shifted towards a more detailed section 4).
> 
> ... I agree there isn't a lot to be gained by bikeshedding 
> over these old tests now, unless there are serious concerns 
> about the validity of the process they employed, that would 
> make them bad examples to follow, for people who wished to do 
> later testing.
> 
> The focus of future editing should mainly be on the relevance 
> and validity of any new tests that people wish to add to it 
> as they come.
> 
> I completely agree that it's highly desirable for us to avoid 
> misleading "the industry" about the characteristics of the 
> final RFC defined codec, so results should only be added 
> there after WG scrutiny and consensus.
> 
> Cheers,
> Ron
> 
> 
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec
> 