Re: [codec] A concrete proposal for requirements and testing

["Roni Even" <[e-mail]>]

Hi,

I would expect that a message that in my view calls for consensus, to come
from the WG chairs. 

 

As for your question, saying that a presentation by one tester addresses the
requirements is not sufficient in my view. I would expect to see a document
that summarized all tests done based on a common test plan by more than one
tester. The problem is that if there is no plan to comment on and to  use
you cannot compare between different results.

I think that Paul sent an example of how to draft a plan that can be used.

 

As for removing GSM-FR, G.722 and Speex-UWB I am OK. As far as I remember
the meeting there was no consensus on the reference codecs.

 

Roni Even

 

 

From: [e-mail] [mailto:[e-mail]] On Behalf Of
Jean-Marc Valin
Sent: Thursday, March 31, 2011 4:54 PM
To: [e-mail]
Subject: [codec] A concrete proposal for requirements and testing

 

Hi,

Following the meeting and post-meeting discussions about requirements and
testing, we would like to make the following proposal which addresses the
opposing views which prevented consensus in the meeting today. 

First, we propose to remove the following codecs from the requirements:

- GSM-FR, based on consensus from the list
- G.722, based on being clearly out-performed by G.722.1
- Speex-UWB, based on the fact that the author himself does not recommend it
being used :-)

We can keep the other reference codecs as minimum quality requirement and
include being no worse than AMR-NB and AMR-WB as "objectives" that are "nice
to have", but not hard requirements.

>From there and based on the listening tests presented by Jan Skoglund today,
let's see what we can already conclude and what still needs more testing:

1) The narrowband test showed that Opus had higher quality than Speex at 11
kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2
requirement of out-performing Speex in narrowband mode?

2) The narrowband test showed that Opus had higher quality at 11 kb/s than
iLBC at 15 kb/s. Does anyone disagree that this is sufficient to meet the
Sec 4.2 requirement of out-performing iLBC.

3) There have been no formal comparison with AMR-NB yet. What do you think
would be sufficient to assess the quality of Opus compared to AMR-NB?

4) The wideband test showed that Opus at 19.85 kb/s had higher quality than
Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient to meet
the Sec 4.2 requirement of out-performing Speex in wideband mode?

5) The wideband test showed that Opus at 19.85 kb/s had higher quality than
G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to meet the
Sec 4.2 requirement of out-performing G.722.1?

6) The wideband test showed that Opus at 19.85 kb/s had higher quality than
AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient to
concluded that the proposed "nice to have" objective of "no worse than
AMR-WB" is met?

7) The fullband test showed that Opus at 32 kb/s had higher quality than
G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet the
Sec 4.2 requirement of out-performing G.722.1C, considering that G.719 has
already been shown to out-perform G.722.1C

If you disagree with any of the points above -- as may very well be the case
-- please do provide a concrete test proposal that would be sufficient to
convince you.

Cheers,

    Jean-Marc and Koen