Re: [codec] A concrete proposal for requirements and testing

[Ron <[e-mail]>]

On Thu, Apr 07, 2011 at 12:36:51PM -0600, Peter Saint-Andre wrote:
> On 4/7/11 12:13 PM, Paul Coverdale wrote:
> 
> > It doesn't matter what any single individual on this mailing list thinks
> > about Opus quality. What matters is the collective opinion of the population
> > of naïve users who may listen to speech/music over Opus in the future, and
> > how they rate it compared to listening to speech/music via other codecs (not
> > that they will know/care what a codec is...). A well-designed test plan and
> > statistical analysis can give a good estimate of this opinion.
> 
> I completely agree. That's why it is so important for us to publish
> draft-ietf-codec-opus as a Proposed Standard.
> 
> The IETF tradition is one of "rough consensus and running code". What
> this means is that we work hard on a new technology and publish it as an
> RFC ("request for comments"). This is the rough consensus part.
> 
> Then we implement it and deploy it. Testing is something that happens
> naturally when we run a technology on networks. This is the "running
> code" part.
> 
> Implementation and deployment experience can lead to revisions and
> refinements and better specification of the technology. Of course we'll
> also seek rough consensus on any modifications, but it's not necessary,
> or even desirable, to get that implementation and deployment experience
> before we publish a specification as a Proposed Standard RFC.
> 
> Indeed, much discussion has occurred of late within the IETF about
> simplifying the Internet Standards Process by making it easier, not
> harder, to advance Internet-Drafts to Proposed Standard:
> 
> https://datatracker.ietf.org/doc/draft-housley-two-maturity-levels/
> 
> Let's get to rough consensus and then start running this code on the
> network. That will be the true test.

+1

Peter speaks well exactly what went through my mind about the best way
to actually get a meaningful statistical analysis of what we have today,
from a real world mix of naïve, golden-eared, and technically savvy users,
over a huge variety of use cases.

Of course I wasn't suggesting that we should delegate Stephen to go have
a listen, then hinge the remainder of this process on his impressions.
I was suggesting that *everyone* proposing tests should go have a listen,
because if at G.711 (mono) data rates, you can't ABX (stereo) Opus from
the reference (as many good listeners apparently can't on many samples),
then clearly many of the tests that people have been proposing aren't
actually very meaningful at all.  It might be nice to have a number that
shows how much we are better than them, to brag about over dinner with
p < 0.05 confidence, but it's already self-evident that we've by far
exceeded the quality goals that we initially merely aspired to aim for.
The sky is blue.  Opus is almost transparent.  Most people who aren't
blind or deaf seem to have rough consensus on these things already.

Audio quality isn't the leading problem we have to solve at this stage.
The biggest problem now is the problems we still don't know about yet.
And the only way to find them is to give it to real users, and see what
they can break.  The developers can't break it anymore, and nobody here
has shown that they can either.  That says it's time to expand our circle
of benevolent wreckers to people with more imaginative (ab)uses than we
are able to contrive here in the lab from molecules of past experience.

Please don't stop testing.  But please don't stop everyone else while
you do.  There are many more people waiting to test this who are outside
this group than we currently have within it.  It's time to put them to
work for us :)  Implement your real use case and ship it.  Nothing else
will show us what bugs really do remain as efficiently as that will.

Cheers,
Ron