Re: [codec] A concrete proposal for requirements and testing

["Benjamin M. Schwartz" <[e-mail]>]

On 04/07/2011 08:50 PM, Roman Shpount wrote:
> Quoting Nokia's paper: "Codecs done without thorough standardization
> effort like Speex and iLBC offer significantly reduced efficiency,
> probably due to much lesser optimization, *listening tests* and IPR free
> design." Testing is an important part of development and standardization
> process for a CODEC.

How do you know when you have done enough development testing to propose a
standard?

You know that you have done enough testing when you perform a battery of
tests, and the results don't make you want to change the specification.

An enormous amount of testing has been performed on Opus, as part of the
rapid development feedback loop.  Those tests are now obsolete, because
each of them provoked changes in the specification to improve
performance.[1]  The latest round of tests from Google, Hydrogen Audio,
and Broadcom total many thousands of quality assessments and a huge
expenditure of time and effort ... and the results had no effect.  None.
All of the Opus maintainers agree that the results do not suggest any
further change to the specification.

That means we have done enough development testing.  Now it is time to
propose a standard, so that we have something on which to perform formal
assessment.

[1] Here's one that shows an ancient version of CELT beating G.722.1C at
48kbps: http://www.celt-codec.org/comparison/