Re: [codec] Testing: A Novel Proposal

[Monty Montgomery <[e-mail]>]

> The results presented so far are good, the actual tests however are debatable.

So far, the people who want to use the codec have been testing it the
way they'd want to use it.  That's not complete use case testing, but
I wager it covers the most likely uses.

> Testing is not delaying the process, it is part of the process.

That is the form of testing that has been and is being performed; lean
and targeted in-process testing that immediately feeds back into
further perfection of the codec at hand.  Results have been used
progressively and immediately.

I can't agree with halting development for years of distributed
testing across dozens of officially accredited labs.  This form of
testing is welcome as well, but it will not produce a better codec. It
will merely provide bullet points for subsequent marketing materials
down the road (as well as inform development of the next generation of
codecs of course; it is a mistake to be too short-sighted.  But it
won't improve Opus.)

>> 4) perfection is the enemy of good
>>
> So is imperfection.

I have an interesting proposal.   By way of grandiose schoolyard
bravado, in the interests of sparking a more productive direction of
debate:

"If it isn't good enough... Prove it."

The group of proponents actively participating in the development of
Opus have test results showing that so far Opus is doing very well
indeed.

A group of skeptics that happen to be outside the development process
are calling for a halt to give time for further evaluation.

Testing is important OF COURSE, but so far the abstract test plans
suggested, as opposed to the tests actually being performed, are calls
for a complete and indefinite halt to progress (with the side effect
that it will lead to a delay of wider real-world testing as favored by
the proponent faction). On which side is the burden of proof at this
point?  The proponents with good data or the more cautious faction
with an abstract five year plan?  Both sides favor testing.  The
skeptics simply want a different process that the proponents see as
directly harmful to progress... and actual testing.

So here's a Novel Idea: Perhaps what we really need to move things
forward is a 'Devil's Advocate' office, a blessed and fully _parallel_
testing 'faction' that is out to make Opus look as bad as possible (in
the name of progress) and has a responsibility of keeping up with
development.  This will be immensely useful.  It also solves the
problem of "I think it isn't good enough, but it's up to you to prove
it for me."

Any takers?  I have a hat with little horns on it.

Monty
(Individual)