Re: [codec] A concrete proposal for requirements and testing

[Jean-Marc Valin <[e-mail]>]

On 11-04-06 06:41 PM, Paul Coverdale wrote:
> When I mentioned 40-60 listeners for an MOS test, this was the number that
> we used to try to use in tests in Nortel. Of course, more listeners are
> better, but often there are constraints on time and money. The ITU examples
> I sent earlier specify 32 listeners for the NB/WB test and 24 for the SWB
> test. I would suggest 24 listeners as a minimum.

Thanks. Noted.

> I (and others) suggested the use of objectives ("nice to have") as well as
> requirements ("must have") in the requirements document, but I didn't
> specifically propose "no worse than AMR-NB" (or AMR-WB) as an objective. In
> fact, my preference would be to include AMR-NB and AMR-WB as requirements in
> section 4.2, since there is not much else of substance there.

I strongly disagree with including AMR-NB and AMR-WB as strict 
requirements for several reasons. Those codecs are not credible 
alternatives to Opus within the charter of this WG, to "try to avoid 
encumbered technologies."

Also, I note from the ITU test plan for G.729.1 (then G.729EV) you sent 
that it does not even mention AMR-NB. As for AMR-WB (G.722.2), it is 
only included at a bit-rate of 8.85 kb/s even though the lowest G.729.1 
bit-rate is (AFAIK) 14 kb/s. Looking at this paper from Nokia: 
http://research.nokia.com/files/public/%5B11%5D_ICASSP2010_Voice%20Quality%20Evaluation%20of%20Various%20Codecs.pdf

we see that G.729.1 is clearly worse than AMR-WB. Despite that, it seems 
like the ITU-T has seen the value of this codec, which supports both 
narrowband and wideband at multiple rates. The codec we're designing 
here is meant to do not only narrowband and wideband, but also 
super-wideband and fullband, for both speech and music and for mono and 
stereo. This is something which -- to the best of my knowledge -- has 
not been done before for an interactive codec. Requiring Opus to also 
beat all codecs out there in all circumstances is thus totally 
unreasonable (despite the fact that the tests we did so far indicate we 
may be close to that).

Cheers,

	Jean-Marc