Re: [codec] A concrete proposal for requirements and testing

My recollection was that everyone supported removing GSM-FR.  All other
removals had strong hums on both sides, so there was clearly no consensus.

Although the anchor codecs matter, I think another topic that needs closure
are the speech conditions.  Usually there is some clean speech, but also
samples with noisy or reverberant speech.  Sometimes multiple speakers are
also tested.

I would also suggest at least one tandeming test.

Steve B.

On Tue, Apr 5, 2011 at 1:54 PM, Roni Even <ron.even.tlv@gmail.com> wrote:

> Hi,
>
> I would expect that a message that in my view calls for consensus, to come
> from the WG chairs.
>
>
>
> As for your question, saying that a presentation by one tester addresses
> the requirements is not sufficient in my view. I would expect to see a
> document that summarized all tests done based on a common test plan by more
> than one tester. The problem is that if there is no plan to comment on and
> to  use you cannot compare between different results.
>
> I think that Paul sent an example of how to draft a plan that can be used.
>
>
>
> As for removing GSM-FR, G.722 and Speex-UWB I am OK. As far as I remember
> the meeting there was no consensus on the reference codecs.
>
>
>
> Roni Even
>
>
>
>
>
> *From:* codec-bounces@ietf.org [mailto:codec-bounces@ietf.org] *On Behalf
> Of *Jean-Marc Valin
> *Sent:* Thursday, March 31, 2011 4:54 PM
> *To:* codec@ietf.org
> *Subject:* [codec] A concrete proposal for requirements and testing
>
>
>
> Hi,
>
> Following the meeting and post-meeting discussions about requirements and
> testing, we would like to make the following proposal which addresses the
> opposing views which prevented consensus in the meeting today.
>
> First, we propose to remove the following codecs from the requirements:
>
> - GSM-FR, based on consensus from the list
> - G.722, based on being clearly out-performed by G.722.1
> - Speex-UWB, based on the fact that the author himself does not recommend
> it being used :-)
>
> We can keep the other reference codecs as minimum quality requirement and
> include being no worse than AMR-NB and AMR-WB as "objectives" that are "nice
> to have", but not hard requirements.
>
> From there and based on the listening tests presented by Jan Skoglund
> today, let's see what we can already conclude and what still needs more
> testing:
>
> 1) The narrowband test showed that Opus had higher quality than Speex at 11
> kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2
> requirement of out-performing Speex in narrowband mode?
>
> 2) The narrowband test showed that Opus had higher quality at 11 kb/s than
> iLBC at 15 kb/s. Does anyone disagree that this is sufficient to meet the
> Sec 4.2 requirement of out-performing iLBC.
>
> 3) There have been no formal comparison with AMR-NB yet. What do you think
> would be sufficient to assess the quality of Opus compared to AMR-NB?
>
> 4) The wideband test showed that Opus at 19.85 kb/s had higher quality than
> Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient to meet
> the Sec 4.2 requirement of out-performing Speex in wideband mode?
>
> 5) The wideband test showed that Opus at 19.85 kb/s had higher quality than
> G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to meet the
> Sec 4.2 requirement of out-performing G.722.1?
>
> 6) The wideband test showed that Opus at 19.85 kb/s had higher quality than
> AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient to
> concluded that the proposed "nice to have" objective of "no worse than
> AMR-WB" is met?
>
> 7) The fullband test showed that Opus at 32 kb/s had higher quality than
> G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet the
> Sec 4.2 requirement of out-performing G.722.1C, considering that G.719 has
> already been shown to out-perform G.722.1C
>
> If you disagree with any of the points above -- as may very well be the
> case -- please do provide a concrete test proposal that would be sufficient
> to convince you.
>
> Cheers,
>
>     Jean-Marc and Koen
>
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec
>
>