Re: [codec] A concrete proposal for requirements and testing

[Jan Skoglund <[e-mail]>]

No, we only had clean speech in our tests.

Jan



On Tue, Apr 5, 2011 at 11:24 AM, Jean-Marc Valin <
[e-mail]> wrote:

> On 11-04-05 02:00 PM, Stephen Botzko wrote:
>
>> My recollection was that everyone supported removing GSM-FR. All other
>> removals had strong hums on both sides, so there was clearly no consensus.
>>
>
> Correct. The only other codecs that were removed (G.722 and Speex-UWB) from
> that *proposal* had no hums on them, but we expected there would be no
> objections.
>
>
>  Although the anchor codecs matter, I think another topic that needs
>> closure
>> are the speech conditions. Usually there is some clean speech, but also
>> samples with noisy or reverberant speech. Sometimes multiple speakers are
>> also tested.
>>
>
> Makes sense. As a note, some of the speech samples in the Google tests were
> using reverberant with (IIRC) some office noise.
>
>
>  I would also suggest at least one tandeming test.
>>
>
> What use case would you suggest that involves tandeming?
>
> Cheers,
>
>        Jean-Marc
>
>
>  Steve B.
>>
>> On Tue, Apr 5, 2011 at 1:54 PM, Roni Even <[e-mail]
>> <mailto:[e-mail]>> wrote:
>>
>>    Hi,
>>
>>    I would expect that a message that in my view calls for consensus, to
>>    come from the WG chairs.
>>
>>    As for your question, saying that a presentation by one tester
>>    addresses the requirements is not sufficient in my view. I would expect
>>    to see a document that summarized all tests done based on a common test
>>    plan by more than one tester. The problem is that if there is no plan
>>    to comment on and to use you cannot compare between different results.
>>
>>    I think that Paul sent an example of how to draft a plan that can be
>> used.
>>
>>    As for removing GSM-FR, G.722 and Speex-UWB I am OK. As far as I
>>    remember the meeting there was no consensus on the reference codecs.
>>
>>    Roni Even
>>
>>    *From:*[e-mail] <mailto:[e-mail]>
>>    [mailto:[e-mail] <mailto:[e-mail]>] *On
>>
>>    Behalf Of *Jean-Marc Valin
>>    *Sent:* Thursday, March 31, 2011 4:54 PM
>>    *To:* [e-mail] <mailto:[e-mail]>
>>
>>    *Subject:* [codec] A concrete proposal for requirements and testing
>>
>>    Hi,
>>
>>    Following the meeting and post-meeting discussions about requirements
>>    and testing, we would like to make the following proposal which
>>    addresses the opposing views which prevented consensus in the meeting
>>    today.
>>
>>    First, we propose to remove the following codecs from the requirements:
>>
>>    - GSM-FR, based on consensus from the list
>>    - G.722, based on being clearly out-performed by G.722.1
>>    - Speex-UWB, based on the fact that the author himself does not
>>    recommend it being used :-)
>>
>>    We can keep the other reference codecs as minimum quality requirement
>>    and include being no worse than AMR-NB and AMR-WB as "objectives" that
>>    are "nice to have", but not hard requirements.
>>
>>     From there and based on the listening tests presented by Jan Skoglund
>>    today, let's see what we can already conclude and what still needs more
>>    testing:
>>
>>    1) The narrowband test showed that Opus had higher quality than Speex
>>    at 11 kb/s. Does anyone disagree that this is sufficient to meet the
>>    Sec 4.2 requirement of out-performing Speex in narrowband mode?
>>
>>    2) The narrowband test showed that Opus had higher quality at 11 kb/s
>>    than iLBC at 15 kb/s. Does anyone disagree that this is sufficient to
>>    meet the Sec 4.2 requirement of out-performing iLBC.
>>
>>    3) There have been no formal comparison with AMR-NB yet. What do you
>>    think would be sufficient to assess the quality of Opus compared to
>> AMR-NB?
>>
>>    4) The wideband test showed that Opus at 19.85 kb/s had higher quality
>>    than Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient
>>    to meet the Sec 4.2 requirement of out-performing Speex in wideband
>> mode?
>>
>>    5) The wideband test showed that Opus at 19.85 kb/s had higher quality
>>    than G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient
>>    to meet the Sec 4.2 requirement of out-performing G.722.1?
>>
>>    6) The wideband test showed that Opus at 19.85 kb/s had higher quality
>>    than AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient
>>    to concluded that the proposed "nice to have" objective of "no worse
>>    than AMR-WB" is met?
>>
>>    7) The fullband test showed that Opus at 32 kb/s had higher quality
>>    than G.719 at 32 kb/s. Does anyone disagree that this is sufficient to
>>    meet the Sec 4.2 requirement of out-performing G.722.1C, considering
>>    that G.719 has already been shown to out-perform G.722.1C
>>
>>    If you disagree with any of the points above -- as may very well be the
>>    case -- please do provide a concrete test proposal that would be
>>    sufficient to convince you.
>>
>>    Cheers,
>>
>>    Jean-Marc and Koen
>>
>>
>>    _______________________________________________
>>    codec mailing list
>>    [e-mail] <mailto:[e-mail]>
>>
>>    https://www.ietf.org/mailman/listinfo/codec
>>
>>
>>
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec
>