Re: [codec] A concrete proposal for requirements and testing

[Jean-Marc Valin <[e-mail]>]

On 11-04-07 08:50 PM, Roman Shpount wrote:
> Quoting Nokia's paper: "Codecs done without thorough standardization 
> effort like Speex and iLBC offer significantly reduced efficiency, 
> probably due to much lesser optimization, *listening tests* and IPR 
> free design." Testing is an important part of development and 
> standardization process for a CODEC.

I don't think I'm qualified to comment on iLBC, but when it comes to 
Speex, I think I know better than anyone why it does not perform as well 
as codecs like AMR. Probably the main reason is quite simple: it was 
written by a PhD student (me) with no prior experience in speech coding 
as a side project in during his spare time. Another reason is the fact 
that the original goal was just to get something decent and free out 
there, not to beat (or even equal) the state-of-the-art. Avoiding 
encumbered technology (e.g. ACELP) also had something to do with it, 
along with the fact that I did almost all the code by myself. Lack of 
listening test at the end of the process was totally irrelevant. Even if 
I had all the listening tests in the world at that point, it wouldn't 
have changed anything.

This is the exact opposite of what has happened so far with Opus. I'm 
obviously much more experienced than when I started Speex in 2002, but 
I'm also no longer working by myself. There have been other developers 
(e.g. Tim and Greg) involved right from the start, and since the codec 
WG was created, there's been a lot of collaboration between all the 
companies/developers involved in Opus. This has let to many improvements 
to the components that were originally the SILK and CELT codecs. The 
last difference is testing. But not the type of testing that's been 
discussed in this thread. Spending a lot of time to just get back a 
handful of numbers is mostly useless in terms of improving a codec. The 
kind of testing that has allowed us to make huge progress is of a 
different kind. More along the lines of "I tried playing live with my 
friends and all the instruments sounded great except for my bass guitar" 
(true story) or "I used this on this dong from my collection and at 
exactly this point in the file I hear this funny artefact". This is the 
kind of testing that has lead to *huge* improvements to Opus.

Cheers,

     Jean-Marc