[codec] Testing, requirements, and IETF process

[Cullen Jennings <[e-mail]>]

Just wanted to comment on a few bits from other threads. 

Requirements, and testing to see if a technology meets those requirements, can serve many purposes. One is to help guide the development and select between alternative proposals. Often we use requirements to drive design decisions. Other times we use testing of requirements to help evaluate if a given codec would meet our needs or not. Other times we use requirements and test results to help a particular implementer or user decide which technology was optimal for a given product they are developing or deploying. I think there are several points that many people would agree with 

1) test results are good

2) understanding how a test was done and being able to reproduce it increases the confidence in test results 

3) we can't delay forever doing testing

4) perfection is the enemy of good

When this WG decides if opus is ready to publish, they key questions will be do people think this specification should be published or not. Clearly to make an informed decision about that many people will want to know things like which requirements does it meet and which does it not meet and what sort of testing has been done and the results of that testing. At that point opus might proceed to become a proposed standard. Note it is *proposed* - after further deployment experience and testing and possible fixes it might become a draft standard and then it could finally become an internet standard after even more work. In parallel to all this, as someone else pointed out, testing will continue and that testing will help inform people about how this compares to other codecs so that can decide in which applications opus might be an appropriate codec to use. 

In the last meeting there was some discussion on they idea that there are key use cases that are important to lots of people, these use cases implied certain requirements, and prioritizing testing of theses requirements was probably a good idea. Some people would be interested in how this compares to codecs that are not royalty free, and I'm sure that test results with non RF codecs will help inform some decisions. However, the WG would like to produce a royalty free codec and is particularly interested in how opus compares royalty free codecs. The question of if opus is royalty free is not a question this WG is debating though the IPR disclosures are noted. The question of how opus compares to royalty free codecs is one that many people see as critical to the decision to publish opus or not.