Re: [codec] A concrete proposal for requirements and testing

[Anisse Taleb <[e-mail]>]

Dear JM,
> >>1) The narrowband test showed that Opus had higher quality than Speex at
> > 11 kb/s. Does anyone disagree that this is sufficient to meet the Sec
> > 4.2 requirement of out-performing Speex in narrowband mode?
> >
> >>2) The narrowband test showed that Opus had higher quality at 11 kb/s
> > than iLBC at 15 kb/s. Does anyone disagree that this is sufficient to
> > meet the Sec 4.2 requirement of out-performing iLBC.
> >
> > The tests conducted by Jan were performed with clean speech only (to the
> > best of my knowledge), in addition the methodology used (MUSHRA) is
> > admittedly limited when it comes to testing narrowband signals and
> > especially speech. Moreover, I am not sure as to the use of different
> > native language listeners. The test is useful in itself, however, more
> > is needed in order to reach the conclusion you claim.
> 
> Thank you for answering my questions. So I understand that what you
> would like to see is a MOS-type listening test with 40-60 listeners
> (from memory, number quoted by Paul Coverdale at the codec dinner) in
> both English and (I assume) Chinese. Is that correct? Other things that
> should be included for narrowband in your opinion?

The languages used for testing will depend on the labs participating in the testing,
I would not explicitly suggest English and Chinese at this point. 

For the number of listeners, we need to balance the cost of testing and the required accuracy of the test and the methodology used. I would not venture in setting the exact number of listeners, but requiring a minimum of say 24 listeners could be reasonable.
 
> 
> >>3) There have been no formal comparison with AMR-NB yet. What do you
> > think would be sufficient to assess the quality of Opus compared to AMR-
> NB?
> >
> > Not sure what the question means, as was discussed during the meeting, I
> > think the inclusion and comparison to state of the art codecs, including
> > AMR-NB, is a must in order to fully assess the quality of Opus.
> 
> In the original email, Koen and I were proposing to include "no worse
> than AMB-NB" as a desirable *objective* (rather than a hard
> requirement), as proposed by Paul Coverdale during the meeting.

My understanding of the concept of "objectives" is similar to that of a "Bonus", i.e. we require the codec to have certain "requirements" and would like, as a "bonus", to have more than just fulfilling the requirements.
As you phrased it, what would be the requirement on Opus wrt. AMR if NWT is the objective?

> 
> >>4) The wideband test showed that Opus at 19.85 kb/s had higher quality
> > than Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient
> > to meet the Sec 4.2 requirement of out-performing Speex in wideband mode?
> >
> >>5) The wideband test showed that Opus at 19.85 kb/s had higher quality
> > than G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to
> > meet the Sec 4.2 requirement of out-performing G.722.1?
> >>6) The wideband test showed that Opus at 19.85 kb/s had higher quality
> > than AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient
> > to concluded that the proposed "nice to have" objective of "no worse
> > than AMR-WB" is met?
> >
> > Same comment as for 1) and 2), higher quality cannot be concluded on a
> > such a restrictive test set.
> 
> So for wideband, my understanding is that you'd also like to see a
> MOS-type listening test with 40-60 listeners in both English and
> Chinese. Is that correct? Anything else on your list?
> 

Correct, also all item types should be considered, i.e.  clean, noisy, reverberant speech with and without interfering talker, music, speech and music mixed.


> >>7) The fullband test showed that Opus at 32 kb/s had higher quality than
> > G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet
> > the Sec 4.2 requirement of out-performing G.722.1C, considering that
> > G.719 has already been shown to out-perform G.722.1C
> >
> > Not strongly against taking such shortcuts, however, please consider
> > that G.719 has been tested with many signals including reverberant
> > speech, background noise conditions etc...
> 
> OK. So I understand that the main issues you have are with narrowband
> and wideband, right?

Some of the issues apply equally on all bandwidths, so the issues are not only for wideband and narrowband.


Kind regards,
/Anisse