Re: [codec] A concrete proposal for requirements and testing

On 11-04-05 02:00 PM, Stephen Botzko wrote:
> My recollection was that everyone supported removing GSM-FR. All other
> removals had strong hums on both sides, so there was clearly no consensus.

Correct. The only other codecs that were removed (G.722 and Speex-UWB) from 
that *proposal* had no hums on them, but we expected there would be no 
objections.

> Although the anchor codecs matter, I think another topic that needs closure
> are the speech conditions. Usually there is some clean speech, but also
> samples with noisy or reverberant speech. Sometimes multiple speakers are
> also tested.

Makes sense. As a note, some of the speech samples in the Google tests were 
using reverberant with (IIRC) some office noise.

> I would also suggest at least one tandeming test.

What use case would you suggest that involves tandeming?

Cheers,

	Jean-Marc

> Steve B.
>
> On Tue, Apr 5, 2011 at 1:54 PM, Roni Even <ron.even.tlv@gmail.com
> <mailto:ron.even.tlv@gmail.com>> wrote:
>
>     Hi,
>
>     I would expect that a message that in my view calls for consensus, to
>     come from the WG chairs.
>
>     As for your question, saying that a presentation by one tester
>     addresses the requirements is not sufficient in my view. I would expect
>     to see a document that summarized all tests done based on a common test
>     plan by more than one tester. The problem is that if there is no plan
>     to comment on and to use you cannot compare between different results.
>
>     I think that Paul sent an example of how to draft a plan that can be used.
>
>     As for removing GSM-FR, G.722 and Speex-UWB I am OK. As far as I
>     remember the meeting there was no consensus on the reference codecs.
>
>     Roni Even
>
>     *From:*codec-bounces@ietf.org <mailto:codec-bounces@ietf.org>
>     [mailto:codec-bounces@ietf.org <mailto:codec-bounces@ietf.org>] *On
>     Behalf Of *Jean-Marc Valin
>     *Sent:* Thursday, March 31, 2011 4:54 PM
>     *To:* codec@ietf.org <mailto:codec@ietf.org>
>     *Subject:* [codec] A concrete proposal for requirements and testing
>
>     Hi,
>
>     Following the meeting and post-meeting discussions about requirements
>     and testing, we would like to make the following proposal which
>     addresses the opposing views which prevented consensus in the meeting
>     today.
>
>     First, we propose to remove the following codecs from the requirements:
>
>     - GSM-FR, based on consensus from the list
>     - G.722, based on being clearly out-performed by G.722.1
>     - Speex-UWB, based on the fact that the author himself does not
>     recommend it being used :-)
>
>     We can keep the other reference codecs as minimum quality requirement
>     and include being no worse than AMR-NB and AMR-WB as "objectives" that
>     are "nice to have", but not hard requirements.
>
>      From there and based on the listening tests presented by Jan Skoglund
>     today, let's see what we can already conclude and what still needs more
>     testing:
>
>     1) The narrowband test showed that Opus had higher quality than Speex
>     at 11 kb/s. Does anyone disagree that this is sufficient to meet the
>     Sec 4.2 requirement of out-performing Speex in narrowband mode?
>
>     2) The narrowband test showed that Opus had higher quality at 11 kb/s
>     than iLBC at 15 kb/s. Does anyone disagree that this is sufficient to
>     meet the Sec 4.2 requirement of out-performing iLBC.
>
>     3) There have been no formal comparison with AMR-NB yet. What do you
>     think would be sufficient to assess the quality of Opus compared to AMR-NB?
>
>     4) The wideband test showed that Opus at 19.85 kb/s had higher quality
>     than Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient
>     to meet the Sec 4.2 requirement of out-performing Speex in wideband mode?
>
>     5) The wideband test showed that Opus at 19.85 kb/s had higher quality
>     than G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient
>     to meet the Sec 4.2 requirement of out-performing G.722.1?
>
>     6) The wideband test showed that Opus at 19.85 kb/s had higher quality
>     than AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient
>     to concluded that the proposed "nice to have" objective of "no worse
>     than AMR-WB" is met?
>
>     7) The fullband test showed that Opus at 32 kb/s had higher quality
>     than G.719 at 32 kb/s. Does anyone disagree that this is sufficient to
>     meet the Sec 4.2 requirement of out-performing G.722.1C, considering
>     that G.719 has already been shown to out-perform G.722.1C
>
>     If you disagree with any of the points above -- as may very well be the
>     case -- please do provide a concrete test proposal that would be
>     sufficient to convince you.
>
>     Cheers,
>
>     Jean-Marc and Koen
>
>
>     _______________________________________________
>     codec mailing list
>     codec@ietf.org <mailto:codec@ietf.org>
>     https://www.ietf.org/mailman/listinfo/codec
>
>