Re: [codec] A concrete proposal for requirements and testing

[Erik Norvell <[e-mail]>]

+1
 
The tests presented so far serve well in aiding the development work, but they are not mature enough to support general conclusions on the Opus performance. I think the examples from Paul are a good starting point for specifying the tests.

Best regards,
Erik
 


________________________________

	From: [e-mail] [mailto:[e-mail]] On Behalf Of Anisse Taleb
	Sent: den 6 april 2011 10:12
	To: Jean-Marc Valin; [e-mail]
	Subject: Re: [codec] A concrete proposal for requirements and testing
	
	

	Dear Jean Marc,

	 

	Thanks for the fruitful discussions we had at the meeting. Regarding the points you raise, I don't think we can conclude on quality yet. 

	 

	 

	>1) The narrowband test showed that Opus had higher quality than Speex at 11 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing Speex in narrowband mode?

	>2) The narrowband test showed that Opus had higher quality at 11 kb/s than iLBC at 15 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing iLBC.
	
	

	The tests conducted by Jan were performed with clean speech only (to the best of my knowledge), in addition the methodology used (MUSHRA) is admittedly limited when it comes to testing narrowband signals and especially speech. Moreover, I am not sure as to the use of different native language listeners.  The test is useful in itself, however, more is needed in order to reach the conclusion you claim. 

	
	>3) There have been no formal comparison with AMR-NB yet. What do you think would be sufficient to assess the quality of Opus compared to AMR-NB?
	
	

	Not sure what the question means, as was discussed during the meeting, I think the inclusion and comparison to state of the art codecs, including AMR-NB, is a must in order to fully assess the quality of Opus. 

	
	>4) The wideband test showed that Opus at 19.85 kb/s had higher quality than Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing Speex in wideband mode?

	>5) The wideband test showed that Opus at 19.85 kb/s had higher quality than G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing G.722.1?
	>6) The wideband test showed that Opus at 19.85 kb/s had higher quality than AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient to concluded that the proposed "nice to have" objective of "no worse than AMR-WB" is met?
	
	

	Same comment as for 1) and 2), higher quality cannot be concluded on a such a restrictive test set.

	
	>7) The fullband test showed that Opus at 32 kb/s had higher quality than G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing G.722.1C, considering that G.719 has already been shown to out-perform G.722.1C
	
	

	Not strongly against taking such shortcuts, however, please consider that G.719 has been tested with many signals including reverberant speech, background noise conditions etc...

	 

	 

	Kind regards,

	/Anisse

	 

	 

	From: [e-mail] [mailto:[e-mail]] On Behalf Of Jean-Marc Valin
	Sent: Thursday, March 31, 2011 4:54 PM
	To: [e-mail]
	Subject: [codec] A concrete proposal for requirements and testing

	 

	Hi,
	
	Following the meeting and post-meeting discussions about requirements and testing, we would like to make the following proposal which addresses the opposing views which prevented consensus in the meeting today. 
	
	First, we propose to remove the following codecs from the requirements:
	
	- GSM-FR, based on consensus from the list
	- G.722, based on being clearly out-performed by G.722.1
	- Speex-UWB, based on the fact that the author himself does not recommend it being used :-)
	
	We can keep the other reference codecs as minimum quality requirement and include being no worse than AMR-NB and AMR-WB as "objectives" that are "nice to have", but not hard requirements.
	
	From there and based on the listening tests presented by Jan Skoglund today, let's see what we can already conclude and what still needs more testing:
	
	1) The narrowband test showed that Opus had higher quality than Speex at 11 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing Speex in narrowband mode?

	 

	
	
	2) The narrowband test showed that Opus had higher quality at 11 kb/s than iLBC at 15 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing iLBC.
	
	3) There have been no formal comparison with AMR-NB yet. What do you think would be sufficient to assess the quality of Opus compared to AMR-NB?
	
	4) The wideband test showed that Opus at 19.85 kb/s had higher quality than Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing Speex in wideband mode?
	
	5) The wideband test showed that Opus at 19.85 kb/s had higher quality than G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing G.722.1?
	
	6) The wideband test showed that Opus at 19.85 kb/s had higher quality than AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient to concluded that the proposed "nice to have" objective of "no worse than AMR-WB" is met?
	
	7) The fullband test showed that Opus at 32 kb/s had higher quality than G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing G.722.1C, considering that G.719 has already been shown to out-perform G.722.1C
	
	If you disagree with any of the points above -- as may very well be the case -- please do provide a concrete test proposal that would be sufficient to convince you.
	
	Cheers,
	
	    Jean-Marc and Koen