Re: [codec] Testing: A Novel Proposal

[Monty Montgomery <[e-mail]>]

> It was never the intention to make this an impossible

I didn't think so-- but the inadvertant result was initially a
proposal from an expert that was statistically unwinnable as written.
That caused a profound double-take for several people in the Opus
development camp.   I hadn't yet seen a strong statement on the list
of 'Oops, that was thoroughly unintended' as opposed to several 'oh,
pffft, it's not that bad'.  It was that bad, and when the problems
were downplayed on the list, developers prepared for a fight.

> test/task neither was the intention to delay the publication of this codec.

Thank you for making that clear.  Despite trying to follow the
discussion closely, I was also certain there has been an explicit call
for a halt to development (or publication) while testing was planned
and performed.  I am glad to hear there was no such suggestion.

> I am disappointed that some of the codec proponents and supporters are defensive and quite dismissive.

Several of the codec proponents currently feel the same way about the
skeptics.  More than one has privately expressed frustration that
their testing work and results were being casually dismissed.

> This is an opportunity for everyone to show and demonstrate the quality of the codec in a hopefully well designed test that could be reproduced at will and hence not be easily dismissed or argued against.

More than a few participants believed that this was a call to stop
codec work to run a large, unwinnable test suite to be passed before
proceeding any further.  If this is to be characterization testing,
and it's to be performed in parallel with publication and further
work, that is an entirely different situation.  I am glad you have set
the matter straight.

> When it comes to your proposal,
>
> "If it isn't good enough... Prove it."
>
> I do not think I have anything to say about that except perhaps to remind you of your own statements 2 years ago, in response to Roni,
>
> "a lot of characterization, testing, and documentation needs to be done. We're aware of that and we have a lot of work to do."

The time to formally characterize is coming but not yet here; it is
not wrong to be planning, especially as this appears to be your
motivated interest. I would not dream of holding you back, but this
has certainly started off on the wrong foot.  We'll get past it.  I
still think the Devil's Advocate suggestion has a place.  For one
thing, in a DA scheme the skeptics would not appear to be so
completely disengaged from development; appearances shape reality and
should be managed.

In-process feedback testing has been continuous for several years
before the input groups came to the WG, and has ramped up considerably
since the WG formed.

Documentation-- ah, well, we can always do a better job of that.  That
said, I had not noticed any detailed discussion of the spec from
outside the core development group-- indeed this is my primary concern
at the current stage of development, and I have been a bit flummoxed
that post-final characterization testing seemed to be a subject of
urgent debate when detailed spec input and review is far more
immediately important.

As a final observation from someone who manages some of the Opus
developers: The implication on the list that the developers have not
been hard at work and producing code in an intelligent fashion ruffled
some feathers to put it mildly.   In addition, we are and have been
concerned about a lack of detailed, concrete review and debate on the
finer technical points of the proposed codec [as opposed to the time
spent reopening benchmarks supposedly settled prior to the WG
charter].

Monty