Re: [codec] A concrete proposal for requirements and testing

Roman Shpount wrote 
> What I am trying to say is if there is a clear test plan I, and probably a lot of other people on this list, would be able to conduct more tests of the CODECs. 

And we all welcome such tests. The more the better. 

It's just unclear how or why such testing should have any bearing on the WG. There are no other candidates to compete against; it won't retroactively improve the codec; quality has been shown to be good enough for the codec to be useful... 

Perhaps we can set up a separate mailing list for those who want to do ITU-T style codec testing? 

best, 
koen. 

----- Original Message -----
From: "Roman Shpount" <roman@telurix.com> 
To: "Gregory Maxwell" <gmaxwell@juniper.net> 
Cc: codec@ietf.org, "Stephen Botzko" <stephen.botzko@gmail.com> 
Sent: Thursday, April 7, 2011 9:22:21 PM 
Subject: Re: [codec] A concrete proposal for requirements and testing 

Please, don't get me misunderstood: What I am trying to say is if there is a clear test plan I, and probably a lot of other people on this list, would be able to conduct more tests of the CODECs. Not having a test plan makes it a lot more difficult for us to do tests. 
_____________ 
Roman Shpount 

On Thu, Apr 7, 2011 at 10:45 PM, Gregory Maxwell < gmaxwell@juniper.net > wrote: 

Roman Shpount [ roman@telurix.com ] wrote: 
> It is in everybody's interest to produce the best CODEC possible. Comments that "we listen to it and we like it" and "you are free to run your own tests", even though valid, are not very productive. 

Jean-Marc Valin [ jmvalin@jmvalin.ca ] wrote: 
> The kind of testing that has allowed us to make huge progress is of a different kind. 

Someone who hasn't been following all of the discussion but read Jean-Marc's and Roman's posts might reach an erroneous conclusion that controlled blind tests have not been conducted at all. 

But this isn't the case, The "Spending a lot of time to just get back a handful of numbers" tests of the controlled and blind large-panel type have also been conducted— not because they're especially useful in development, but because they are sanity checks, because they are useful to systems integrators and others in understanding where the codec fits, etc. 

The most recent and applicable of these tests was Jan Skoglund's, which included some 346 listener*samples across a variety of relevant codecs and configurations ( http://www.ietf.org/mail-archive/web/codec/current/msg02272.html ) 

Jean-Marc's description of the really helpful™ testing also may have made it sound rather unscientific. Much of the work in development-tuning is finding the interesting cases and that simply works best if you use the codec in a lot of ways. Anything that gets in the way of usage volume is probably bad. After finding them the actual validation of the changes can be done with simple blind A/B testing with a small number of listeners (sometimes just the person doing the tuning themself). There have been hundreds of these tests conducted during development— Enough that one of the persons involved in Opus tuning wrote a new open source tool for blind A/B,A/B/X,X/X/Y testing (Squishyball, http://people.xiph.org/~xiphmont/demo/celt/demo.html#demo ) because the available tools were intrusive and got in the way of high volume testing. 

_______________________________________________ 
codec mailing list 
codec@ietf.org 
https://www.ietf.org/mailman/listinfo/codec 