Re: [codec] A concrete proposal for requirements and testing

[Roman Shpount <[e-mail]>]

Please, don't get me misunderstood: What I am trying to say is if there is a
clear test plan I, and probably a lot of other people on this list, would be
able to conduct more tests of the CODECs. Not having a test plan makes it a
lot more difficult for us to do tests.
_____________
Roman Shpount


On Thu, Apr 7, 2011 at 10:45 PM, Gregory Maxwell <[e-mail]>wrote:

> Roman Shpount [[e-mail]] wrote:
> > It is in everybody's interest to produce the best CODEC possible.
> Comments that "we listen to it and we like it" and "you are free to run your
> own tests", even though valid, are not very productive.
>
> Jean-Marc Valin [[e-mail]] wrote:
> > The kind of testing that has allowed us to make huge progress is of a
> different kind.
>
> Someone who hasn't been following all of the discussion but read
> Jean-Marc's and Roman's posts might reach an erroneous conclusion that
> controlled blind tests have not been conducted at all.
>
> But this isn't the case, The  "Spending a lot of time to just get back a
> handful of numbers" tests of the controlled and blind large-panel type have
> also been conducted— not because they're especially useful in development,
> but because they are sanity checks, because they are useful to systems
> integrators and others in understanding where the codec fits, etc.
>
> The most recent and applicable of these tests was Jan Skoglund's, which
> included some 346 listener*samples across a variety of relevant codecs and
> configurations (
> http://www.ietf.org/mail-archive/web/codec/current/msg02272.html)
>
> Jean-Marc's description of the really helpful™ testing also may have made
> it sound rather unscientific.  Much of the work in development-tuning is
> finding the interesting cases and that simply works best if you use the
> codec in a lot of ways.  Anything that gets in the way of usage volume is
> probably bad.  After finding them the actual validation of the changes can
> be done with simple blind A/B testing with a small number of listeners
> (sometimes just the person doing the tuning themself). There have been
> hundreds of these tests conducted during development— Enough that one of the
> persons involved in Opus tuning wrote a new open source tool for blind
> A/B,A/B/X,X/X/Y testing (Squishyball,
> http://people.xiph.org/~xiphmont/demo/celt/demo.html#demo) because the
> available tools were intrusive and got in the way of high volume testing.
>
>
>