Re: [codec] A concrete proposal for requirements and testing

[Stephen Botzko <[e-mail]>]

In-line-
BR, Stephen

On Wed, Apr 6, 2011 at 4:57 PM, Jean-Marc Valin <[e-mail]> wrote:

>  On 11-04-05 02:00 PM, Stephen Botzko wrote:
>
> Although the anchor codecs matter, I think another topic that needs closure
> are the speech conditions.  Usually there is some clean speech, but also
> samples with noisy or reverberant speech.
>
>
> What's your suggestion for this? Making one set of tests on clean
> conditions and one on noisy+reverberant?
>
> Yes.


>
> Sometimes multiple speakers are also tested.
>
>
> What do you mean by "multiple speakers". All the tests we've done so far
> have used a different speaker for each sample. Or do you mean something
> else?
>
> I was meaning a test with an interfering talker (as shown on page 13 of the
"Quality Assessment Characterisation/Optimisation step1 Test Plan for
the ITU-T G.729 based ..." document that Paul Coverdale posted).  Though the
other background noise conditions on that page are also worth testing I
think.

>
> I would also suggest at least one tandeming test.
>
>
> What form do you see this taking?
>
> You could test OPUS -> AMR-NB with G.711 -> AMR->NB as the anchor (or G.729
-> AMR-NB) with (for instance) noisy speech.
and/or a similar test using OPUS -> AMR-WB, perhaps with G.719-> AMR-WB as
the anchor.


>    Jean-Marc
>
>
>
> Steve B.
>
> On Tue, Apr 5, 2011 at 1:54 PM, Roni Even <[e-mail]> wrote:
>
>>  Hi,
>>
>> I would expect that a message that in my view calls for consensus, to come
>> from the WG chairs.
>>
>>
>>
>> As for your question, saying that a presentation by one tester addresses
>> the requirements is not sufficient in my view. I would expect to see a
>> document that summarized all tests done based on a common test plan by more
>> than one tester. The problem is that if there is no plan to comment on and
>> to  use you cannot compare between different results.
>>
>> I think that Paul sent an example of how to draft a plan that can be used.
>>
>>
>>
>> As for removing GSM-FR, G.722 and Speex-UWB I am OK. As far as I remember
>> the meeting there was no consensus on the reference codecs.
>>
>>
>>
>> Roni Even
>>
>>
>>
>>
>>
>> *From:* [e-mail] [mailto:[e-mail]] *On Behalf
>> Of *Jean-Marc Valin
>> *Sent:* Thursday, March 31, 2011 4:54 PM
>> *To:* [e-mail]
>> *Subject:* [codec] A concrete proposal for requirements and testing
>>
>>
>>
>> Hi,
>>
>> Following the meeting and post-meeting discussions about requirements and
>> testing, we would like to make the following proposal which addresses the
>> opposing views which prevented consensus in the meeting today.
>>
>> First, we propose to remove the following codecs from the requirements:
>>
>> - GSM-FR, based on consensus from the list
>> - G.722, based on being clearly out-performed by G.722.1
>> - Speex-UWB, based on the fact that the author himself does not recommend
>> it being used :-)
>>
>> We can keep the other reference codecs as minimum quality requirement and
>> include being no worse than AMR-NB and AMR-WB as "objectives" that are "nice
>> to have", but not hard requirements.
>>
>> From there and based on the listening tests presented by Jan Skoglund
>> today, let's see what we can already conclude and what still needs more
>> testing:
>>
>> 1) The narrowband test showed that Opus had higher quality than Speex at
>> 11 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2
>> requirement of out-performing Speex in narrowband mode?
>>
>> 2) The narrowband test showed that Opus had higher quality at 11 kb/s than
>> iLBC at 15 kb/s. Does anyone disagree that this is sufficient to meet the
>> Sec 4.2 requirement of out-performing iLBC.
>>
>> 3) There have been no formal comparison with AMR-NB yet. What do you think
>> would be sufficient to assess the quality of Opus compared to AMR-NB?
>>
>> 4) The wideband test showed that Opus at 19.85 kb/s had higher quality
>> than Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient to
>> meet the Sec 4.2 requirement of out-performing Speex in wideband mode?
>>
>> 5) The wideband test showed that Opus at 19.85 kb/s had higher quality
>> than G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to
>> meet the Sec 4.2 requirement of out-performing G.722.1?
>>
>> 6) The wideband test showed that Opus at 19.85 kb/s had higher quality
>> than AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient to
>> concluded that the proposed "nice to have" objective of "no worse than
>> AMR-WB" is met?
>>
>> 7) The fullband test showed that Opus at 32 kb/s had higher quality than
>> G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet the
>> Sec 4.2 requirement of out-performing G.722.1C, considering that G.719 has
>> already been shown to out-perform G.722.1C
>>
>> If you disagree with any of the points above -- as may very well be the
>> case -- please do provide a concrete test proposal that would be sufficient
>> to convince you.
>>
>> Cheers,
>>
>>     Jean-Marc and Koen
>>
>> _______________________________________________
>> codec mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/codec
>>
>>
>
> _______________________________________________
> codec mailing [e-mail]://www.ietf.org/mailman/listinfo/codec
>
>
>