Re: [codec] A concrete proposal for requirements and testing

[Paul Coverdale <[e-mail]>]

Hi Jean-Marc,

Just to clarify a couple of points.

When I mentioned 40-60 listeners for an MOS test, this was the number that
we used to try to use in tests in Nortel. Of course, more listeners are
better, but often there are constraints on time and money. The ITU examples
I sent earlier specify 32 listeners for the NB/WB test and 24 for the SWB
test. I would suggest 24 listeners as a minimum.

I (and others) suggested the use of objectives ("nice to have") as well as
requirements ("must have") in the requirements document, but I didn't
specifically propose "no worse than AMR-NB" (or AMR-WB) as an objective. In
fact, my preference would be to include AMR-NB and AMR-WB as requirements in
section 4.2, since there is not much else of substance there.

Regards,

...Paul


-----Original Message-----
From: [e-mail] [mailto:[e-mail]] On Behalf Of
Jean-Marc Valin
Sent: Wednesday, April 06, 2011 2:32 PM
To: Anisse Taleb
Cc: [e-mail]
Subject: Re: [codec] A concrete proposal for requirements and testing

Hi Anisse,

On 11-04-06 04:11 AM, Anisse Taleb wrote:
>>1) The narrowband test showed that Opus had higher quality than Speex at
> 11 kb/s. Does anyone disagree that this is sufficient to meet the Sec
> 4.2 requirement of out-performing Speex in narrowband mode?
>
>>2) The narrowband test showed that Opus had higher quality at 11 kb/s
> than iLBC at 15 kb/s. Does anyone disagree that this is sufficient to
> meet the Sec 4.2 requirement of out-performing iLBC.
>
> The tests conducted by Jan were performed with clean speech only (to the
> best of my knowledge), in addition the methodology used (MUSHRA) is
> admittedly limited when it comes to testing narrowband signals and
> especially speech. Moreover, I am not sure as to the use of different
> native language listeners. The test is useful in itself, however, more
> is needed in order to reach the conclusion you claim.

Thank you for answering my questions. So I understand that what you 
would like to see is a MOS-type listening test with 40-60 listeners 
(from memory, number quoted by Paul Coverdale at the codec dinner) in 
both English and (I assume) Chinese. Is that correct? Other things that 
should be included for narrowband in your opinion?

>>3) There have been no formal comparison with AMR-NB yet. What do you
> think would be sufficient to assess the quality of Opus compared to
AMR-NB?
>
> Not sure what the question means, as was discussed during the meeting, I
> think the inclusion and comparison to state of the art codecs, including
> AMR-NB, is a must in order to fully assess the quality of Opus.

In the original email, Koen and I were proposing to include "no worse 
than AMB-NB" as a desirable *objective* (rather than a hard 
requirement), as proposed by Paul Coverdale during the meeting.

>>4) The wideband test showed that Opus at 19.85 kb/s had higher quality
> than Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient
> to meet the Sec 4.2 requirement of out-performing Speex in wideband mode?
>
>>5) The wideband test showed that Opus at 19.85 kb/s had higher quality
> than G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to
> meet the Sec 4.2 requirement of out-performing G.722.1?
>>6) The wideband test showed that Opus at 19.85 kb/s had higher quality
> than AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient
> to concluded that the proposed "nice to have" objective of "no worse
> than AMR-WB" is met?
>
> Same comment as for 1) and 2), higher quality cannot be concluded on a
> such a restrictive test set.

So for wideband, my understanding is that you'd also like to see a 
MOS-type listening test with 40-60 listeners in both English and 
Chinese. Is that correct? Anything else on your list?

>>7) The fullband test showed that Opus at 32 kb/s had higher quality than
> G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet
> the Sec 4.2 requirement of out-performing G.722.1C, considering that
> G.719 has already been shown to out-perform G.722.1C
>
> Not strongly against taking such shortcuts, however, please consider
> that G.719 has been tested with many signals including reverberant
> speech, background noise conditions etc.

OK. So I understand that the main issues you have are with narrowband 
and wideband, right?

Cheers,

	Jean-Marc
_______________________________________________
codec mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/codec