Re: [codec] A concrete proposal for requirements and testing

[Gregory Maxwell <[e-mail]>]

Peter Saint-Andre [[e-mail]] wrote:
> I completely agree. That's why it is so important for us to publish
> draft-ietf-codec-opus as a Proposed Standard.
> The IETF tradition is one of "rough consensus and running code". What
> this means is that we work hard on a new technology and publish it as an
> RFC ("request for comments"). This is the rough consensus part.
> Then we implement it and deploy it. Testing is something that happens
> naturally when we run a technology on networks. This is the "running
> code" part.
> Implementation and deployment experience can lead to revisions and
> refinements and better specification of the technology. Of course we'll
> also seek rough consensus on any modifications, but it's not necessary,
>or even desirable, to get that implementation and deployment experience
> before we publish a specification as a Proposed Standard RFC.
[snip]
> Let's get to rough consensus and then start running this code on the
> network. That will be the true test.

Peter's message succinctly describes much of what I've been thinking but having a difficult time expressing clearly.
Thank you.

High effort post-hoc characterization is very useful for "marketing" and somewhat useful for "best practices", but it doesn't appear to be very useful at driving forward a collaborative development process.  We've seen this already: by the time the developers first saw Broadcom's listening test results, we'd already significantly improved the codec in at least one of the two cases where their test showed lower performance,  mostly based on the feedback of some people on hydrogenaudio trying out the codec and reporting weaknesses.  Likewise, we've received considerable computational complexity and memory usage feedback from people trying to implement the codec on _real_ systems and encountering _real_ difficulties, and I think that this is far more useful than anything we could hope to achieve from a prolonged debate about abstract WMOPS figures.

In order to expand the testing further we need to move beyond testing only parts, and we need to start creating some interoperability rather than having everyone using different incompatible development bitstreams.

Anisse Taleb [[e-mail]] wrote:
> Dear Ron,
> There are many individuals and organizations that may be willing to help the testing effort on a voluntary basis provided there is a test plan that is agreed and represents a consensus of the group of what is to be tested.
> The results we have are incomplete, there are many cases which needs to be tested and we are trying to build consensus over those and derive a test plan that can be agreed and be conducted by, hopefully, more than one organization.
> Undermining this WG testing effort doesn't encourage organizations to chip in their resources and technical expertise to make this work go smoothly and according to plan.

Many people here are not especially picky about what testing is performed by other people.  Personally, I invite all comers to perform whatever testing they find interesting and helpful for their own purposes.  I'm sure the working group—as a whole—will find almost any results interesting.  Different groups have different requirements.  For example, some people are interested in Opus' performance in non-interactive applications. These are outside of the scope of the working group, but the results are still interesting.  Other people care about low bitrate performance, which I think is of less interest to many people here, or stereo support, which is interesting to a different and mostly non-overlapping set of people.  I expect that many groups will do the characterization which they find to be most applicable to their interests, and they'll report back to the working group.  I do not see why much consensus is required in advance of this kind of testing—except on the basis of ensuring basic scientific method (e.g. a non-blind test would likely be uninteresting to everyone), which we would probably find that we already agree on without even having a discussion.

What I find more concerning is this late in the process effort to raise the _requirements_: from requiring the WG product to be better than the nearest alternatives according to the WG charter (royalty free-ish things), to requiring it to be better than "best of breed" royalty-bearing codecs in each of the several broadly defined operating areas.  I believe this change is contrary to the intended purpose of the working group, as the charter acknowledges the existence of alternative standards which do not meet the intended licensing goals several times but never proposes the our output must have higher listening test performance then these codecs. Accepting this requirement risks the paradoxical outcome of being unable to finish a codec even if we have one which meets the explicit purpose of the WG's creation. Not being able to _guarantee_ a result, as a reality of the law, is not the same as willfully failing to achieve that result— so I don't find the argument that we may fail compelling.  The charter is abundantly clear about the purpose of the working group and I think it doesn't make any sense to change from optimizing for cases where the WG's purpose is successful to optimizing for cases where the working group has failed especially when such an optimization would increase the chance of failure!

In particular, I think the initial requirements were establishing a level below which collaborative development wouldn't work. If we put out something which underperformed Speex, for example, few people would deploy it and we'd gain little operational feedback from which we could further improve the codec and/or implementations. But at least beating Speex would ensure rapid deployment from at least the segment which is already using Speex and provide the operational feedback required to collaboratively advance the codec, as already we've seen from some users who were using Speex and who could tolerate a non-frozen non-interoperable codec.

By conflating two issues—characterization targets and requirements—we've taken what should be two fairly uncontroversial issues and converted it into a contentious one.  The testing should be uncontroversial because absent a requirement gate the working group will probably support whatever testing the participants find interesting and are able to perform, and the requirements issue should be uncontroversial because there are relatively few codecs which come anywhere near the WG's stated purpose.