Re: [codec] A concrete proposal for requirements and testing

[Jean-Marc Valin <[e-mail]>]

Hi Roni,

Sorry if this sounded like a call for consensus since that's not what it 
was meant to be. Rather, it was merely a *proposal* to get the discussion 
started. More precisely, this is about what needs to be answered. This is 
why we listed these 7 questions in the original email. Taking just the 
first one: "Does anyone disagree that this is sufficient to meet the Sec 
4.2 requirement of out-performing Speex in narrowband mode?", I'm fine if 
you disagree, but if that's the case I'd at least know what test it would 
take to convince you.

Cheers,

	Jean-Marc


On 11-04-05 01:54 PM, Roni Even wrote:
> I would expect that a message that in my view calls for consensus, to come
> from the WG chairs.
>
> As for your question, saying that a presentation by one tester addresses
> the requirements is not sufficient in my view. I would expect to see a
> document that summarized all tests done based on a common test plan by more
> than one tester. The problem is that if there is no plan to comment on and
> to use you cannot compare between different results.
>
> I think that Paul sent an example of how to draft a plan that can be used.
>
> As for removing GSM-FR, G.722 and Speex-UWB I am OK. As far as I remember
> the meeting there was no consensus on the reference codecs.
>
> Roni Even
>
> *From:*[e-mail] [mailto:[e-mail]] *On Behalf Of
> *Jean-Marc Valin
> *Sent:* Thursday, March 31, 2011 4:54 PM
> *To:* [e-mail]
> *Subject:* [codec] A concrete proposal for requirements and testing
>
> Hi,
>
> Following the meeting and post-meeting discussions about requirements and
> testing, we would like to make the following proposal which addresses the
> opposing views which prevented consensus in the meeting today.
>
> First, we propose to remove the following codecs from the requirements:
>
> - GSM-FR, based on consensus from the list
> - G.722, based on being clearly out-performed by G.722.1
> - Speex-UWB, based on the fact that the author himself does not recommend
> it being used :-)
>
> We can keep the other reference codecs as minimum quality requirement and
> include being no worse than AMR-NB and AMR-WB as "objectives" that are
> "nice to have", but not hard requirements.
>
>  From there and based on the listening tests presented by Jan Skoglund
> today, let's see what we can already conclude and what still needs more
> testing:
>
> 1) The narrowband test showed that Opus had higher quality than Speex at 11
> kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2
> requirement of out-performing Speex in narrowband mode?
>
> 2) The narrowband test showed that Opus had higher quality at 11 kb/s than
> iLBC at 15 kb/s. Does anyone disagree that this is sufficient to meet the
> Sec 4.2 requirement of out-performing iLBC.
>
> 3) There have been no formal comparison with AMR-NB yet. What do you think
> would be sufficient to assess the quality of Opus compared to AMR-NB?
>
> 4) The wideband test showed that Opus at 19.85 kb/s had higher quality than
> Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient to meet
> the Sec 4.2 requirement of out-performing Speex in wideband mode?
>
> 5) The wideband test showed that Opus at 19.85 kb/s had higher quality than
> G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to meet
> the Sec 4.2 requirement of out-performing G.722.1?
>
> 6) The wideband test showed that Opus at 19.85 kb/s had higher quality than
> AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient to
> concluded that the proposed "nice to have" objective of "no worse than
> AMR-WB" is met?
>
> 7) The fullband test showed that Opus at 32 kb/s had higher quality than
> G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet the
> Sec 4.2 requirement of out-performing G.722.1C, considering that G.719 has
> already been shown to out-perform G.722.1C
>
> If you disagree with any of the points above -- as may very well be the
> case -- please do provide a concrete test proposal that would be sufficient
> to convince you.
>
> Cheers,
>
> Jean-Marc and Koen
>