Re: [codec] A concrete proposal for requirements and testing

Dear Ron,

There are many individuals and organizations that may be willing to help the testing effort on a voluntary basis provided there is a test plan that is agreed and represents a consensus of the group of what is to be tested.  
The results we have are incomplete, there are many cases which needs to be tested and we are trying to build consensus over those and derive a test plan that can be agreed and be conducted by, hopefully, more than one organization.

Undermining this WG testing effort doesn't encourage organizations to chip in their resources and technical expertise to make this work go smoothly and according to plan. 

Kind regards,
/Anisse

> -----Original Message-----
> From: codec-bounces@ietf.org [mailto:codec-bounces@ietf.org] On Behalf Of
> Ron
> Sent: Thursday, April 07, 2011 6:48 PM
> To: Stephen Botzko
> Cc: codec@ietf.org
> Subject: Re: [codec] A concrete proposal for requirements and testing
> 
> 
> Hi Stephen,
> 
> On Thu, Apr 07, 2011 at 11:46:05AM -0400, Stephen Botzko wrote:
> > Hi Ron
> >
> > This is not a debugging task, it is a codec characterization/quality
> > assessment that will be done on the finished product.
> 
> Yes, I quite understand the formal, dare I say bureaucratic, process that
> is being proposed here.  And whatever you call it, I do wholly respect the
> scientific method that underpins it (and that was used to create and tune
> this codec for that matter :).
> 
> But I like reality too.  And the reality is:  Have you listened to Opus?
> 
> Is there actually any doubt in your mind that this is not a wholly adequate
> (to understate the case) codec for the purpose proposed, with a few minor
> nits still to be worked out?
> 
> Since we don't actually have another candidate proposed for the WG to
> endorse, what exactly would we be shooting off against?  The quality of
> the codec seems among the least of our worries on the present facts.
> We only have to be better than iLBC to be better than the IETF's current
> best of breed, and nobody has suggested setting the bar that low.
> 
> Which isn't to say that I'm against further quality testing.  But I think
> we have enough data to comfortably say it's Good Enough.  Better Than Most
> even.  I don't think this should be a blocker for the group moving forward
> to assess its other aims and publish the decoder spec.
> 
> Quality improvements can still be made in the encoder after that.
> 
> > In my view, the proper way to go about it is to first get consensus on
> the
> > tests that need to be run, what results are needed, and how the tests
> need
> > to be conducted in order to meaningfully compare the results.  Then get
> > folks signed up to actually do the tests.
> 
> I think you've just described a near impossible task.  This group is far
> too disparate to share a tractable common set of requirements.
> Interested parties should test their own use cases - and in the event of
> trouble, report on those tests with the scientifically reproducible detail
> that you describe.  So others can analyse them, and devise solutions.
> 
> We can equally build consensus on which of those reports are significant
> and which are 'outliers' to the group charter.
> 
> > This is not about collaboration vs competition, rather it is about
> running
> > distributed tests with results that can integrated and scientifically
> > analyzed/compared.  And recording the test methods, in order to allow
> other
> > people in the future to re-do the tests and duplicate the results.
> 
> Well I'd call that collaboration, but I'm not going to compete over what
> to call it.  My point is, doing that stuff is great.  And doing even more
> of it is greater.  But I don't think it needs to hold us up from moving on
> given the results we already have.
> 
> What test would Opus have to not pass for you to think it is unsuitable?
> 
> Best,
> Ron
> 
> 
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec