Re: [codec] Summary of test results

Hi,

Erik comments are valid. We should make some tests on the final codec
specification before WGLC. I mean, in the latest listening tests which used
Opus taken from GIT February 16th, 2011, I found the statement that "with
lower bitrates the codec crashes".  

I mean, just checking the implementation against major programming mistakes
is not a bad think. And this should not be made by the codec designers
themselves.

With best regards,

 Christian

> -----Original Message-----
> From: codec-bounces@ietf.org [mailto:codec-bounces@ietf.org] On Behalf
> Of Koen Vos
> Sent: Wednesday, June 22, 2011 3:32 PM
> To: Erik Norvell
> Cc: codec@ietf.org
> Subject: Re: [codec] Summary of test results
> 
> Hi Erik,
> 
> > However, the only way to make correct statements about the performance
> > of the final Opus codec is to test this final codec.
> 
> We took the MPEG approach to standardization (like with MP3/AAC), which
> means that the codec is defined by its bit-stream rather than its
bit-exact
> behavior.  For this reason, the version tested from February is the final
Opus.
> Besides, the encoder hasn't actually changed in any meaningful way since
> February.
> 
> best,
> koen.
> 
> 
> ----- Original Message -----
> From: "Erik Norvell" <erik.norvell@ericsson.com>
> To: "Jean-Marc Valin" <jean-marc.valin@octasic.com>
> Cc: codec@ietf.org
> Sent: Wednesday, June 22, 2011 2:00:10 AM
> Subject: Re: [codec] Summary of test results
> 
> > -----Original Message-----
> > From: Jean-Marc Valin [mailto:jean-marc.valin@octasic.com]
> > Sent: den 21 juni 2011 21:40
> > To: Erik Norvell
> > Cc: codec@ietf.org
> > Subject: Re: [codec] Summary of test results
> >
> > On 11-06-21 09:04 AM, Erik Norvell wrote:
> > > Thank you for compiling this summary of pre-Opus tests. It should
> > > definitely help in designing the listening test on the final Opus.
> >
> > Just to clarify, the Opus bit-stream *is* final and, as far as these
> > tests (for both speech and music) are concerned, has been since
> > February.
> > The latest draft also has the final stereo bit-stream for voice, but
> > all the rest is long frozen.
> >
> 
> There are a number of tests which are older than that which are still
> referenced when making statements about Opus performance.
> In addition, a frozen bit-stream is not equal to frozen quality. If the
codec
> itself is still permitted to change its quality may be affected.
> 
> 
> > > One comment to section 3: "While Opus has evolved since these tests
> > > were conducted, the results should be considered as a
> > _lower bound_ on
> > > the quality of the final codec."
> > >
> > > I would like to think that the sum is always greater than
> > it's parts,
> > > but it is definitely possible to make something worse by
> > working on it.
> > > Hence, statements about Opus performance must be based on
> > tests made
> > > on the final codec.
> >
> > Of course it's not a guarantee, but there's definitely value in those
> > tests results in that it's unlikely that everything always worked fine
> > and then we just screwed everything up at the end (if that was the
> > case we would have realised it in the other tests).
> >
> 
> I agree the tests are valuable as quality indicators for the codec by the
time
> they were conducted. However, the only way to make correct statements
> about the performance of the final Opus codec is to test this final codec.
To
> deduce that performance from tests of previous versions is bound to
include
> some amount of speculation.
> 
> Best,
> Erik
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec