Re: [codec] A concrete proposal for requirements and testing

[Paul Coverdale <[e-mail]>]

To me, the “finished product” means the final, frozen, design, complete with all of the inherent trade-offs that may have been made. Formal testing, based on an agreed test plan, can then be carried out to determine whether that design does indeed meet all of the initial requirements.

 

Regards,

 

…Paul

 

From: [e-mail] [mailto:[e-mail]] On Behalf Of Koen Vos
Sent: Thursday, April 07, 2011 8:58 PM
To: Roman Shpount
Cc: [e-mail]; Stephen Botzko
Subject: Re: [codec] A concrete proposal for requirements and testing

 

Roman:

How could what Stephen Botzko describes as "a codec characterization/quality assessment that will be done on the finished product" possibly improve coding efficiency?

best,
koen.



  _____  

From: "Roman Shpount" <[e-mail]>
To: "Koen Vos" <[e-mail]>
Cc: "Stephen Botzko" <[e-mail]>, [e-mail]
Sent: Thursday, April 7, 2011 5:50:06 PM
Subject: Re: [codec] A concrete proposal for requirements and testing

Quoting Nokia's paper: "Codecs done without thorough standardization effort like Speex and iLBC offer significantly reduced efficiency, probably due to much lesser optimization, listening tests and IPR free design." Testing is an important part of development and standardization process for a CODEC.

It is in everybody's interest to produce the best CODEC possible. Comments that "we listen to it and we like it" and "you are free to run your own tests", even though valid, are not very productive. I think the best course possible is to treat testing the same way as we do development. We need to collaborate in putting together a comprehensive test plan, and then in actual testing. I am not sure I can contribute a lot in creating a test plan, but I can definitely contribute in testing if such test plan exists. I think this is a common situation for a lot of people in this group.
_____________
Roman Shpount



On Thu, Apr 7, 2011 at 8:24 PM, Koen Vos <[e-mail]> wrote:

Stephen Botzko wrote:
> This is not a debugging task, it is a codec characterization/quality assessment that will be done on the finished product. 

Does such testing have a realistic chance of revealing something that should alter the course of the WG?  
If not, then the testing can be done separately.
If yes: how precisely?

Is the concern that the codec may not be good enough to be worth publishing, given the other codecs already out there?
--> In the tests so far, Opus has consistently outperformed iLBC, Speex, G.722.1, etc.  To me it seems not realistic to expect that pattern to reverse with more or "better" testing.

Or is the concern that for certain input signals or network conditions the codec produces results that are somehow unacceptable?
--> Opus has gone through all kinds of testing during development.  Feel free to do some more, you don't need test plans or consensus for that.  Remember that there will always be untested cases (speech with dog barking in background over dial up hasn't been tried yet I believe), and that's ok.  

Characterization may be useful, but I don't see why it should be a deliverable of the WG.

best,
koen.



  _____  

From: "Stephen Botzko" <[e-mail]>
To: "Ron" <[e-mail]>
Cc: [e-mail]
Sent: Thursday, April 7, 2011 8:46:05 AM


Subject: Re: [codec] A concrete proposal for requirements and testing

Hi Ron

This is not a debugging task, it is a codec characterization/quality assessment that will be done on the finished product. 

In my view, the proper way to go about it is to first get consensus on the tests that need to be run, what results are needed, and how the tests need to be conducted in order to meaningfully compare the results.  Then get folks signed up to actually do the tests.

This is not about collaboration vs competition, rather it is about running distributed tests with results that can integrated and scientifically analyzed/compared.  And recording the test methods, in order to allow other people in the future to re-do the tests and duplicate the results.

Regards,
Stephen Botzko