Re: [codec] Adopt draft-valin-codec-results as a working group item

[Ron <[e-mail]>]

On Thu, Sep 08, 2011 at 09:54:27AM +0000, Anisse Taleb wrote:
> If I recall right, there was a consensus at the meeting in Quebec that this
> document will be assigned to Christian as editor and that work will progress
> into making this document as good and as complete as possible. Thus, IMHO,
> the document is a good starting point for further editing. When it comes to
> the contents of the document, I think what the industry would like to see is
> a full characterization of the Opus codec and its features as an internet
> codec which is a core objective of this work.

I missed being (tele)present for the Quebec meeting, so I can't comment on
what was agreed there, but I agree this document seems like a good starting
point for a WG item, that should be added to as further test results become
available which are accepted by the group as being meaningful and valid.

> On the other hand, tests performed prior to the adoption (or freeze) of the
> codec do not offer much in terms of characterization since they relate to a
> earlier version of the codec and will not accurately reflect the performance
> of the codec.

I don't think they are being misrepresented as such though, do you?
For CELT at least, the versions of the codec that were tested are still
available, and documenting these tests provides useful historical context
for people who may wish to repeat similar tests on the final codec and
gauge the level of improvement that has been made since this group began
its work on them.

> In summary, section 2) and 3) should be removed

... so I disagree with this as a blanket statement if your only reasoning
is "this documents history, not the present".

I think possibly the SILK tests could be clarified to identify exactly
which version of SILK was used for them.  And if you have reason to doubt
the validity of individual tests for reasons other than "they are old",
then that should be considered.  But otherwise, I think these sections
provide useful historical information of both the improvements that were
made and the process of testing that guided them, and that the value and
transparency of this document would not be improved by removing them.

One of the aims of this group was to demonstrate the benefits of using
an open and evolutionary methodology to create something better than
could be achieved by a "develop behind closed doors, winner takes all
shootout" process.  I think documenting the history of testing, and the
results that were seen, is a valuable part of describing that process
and its results for the people who will come after us, and that simply
discarding it now that it has already been compiled would only do them
a disservice.

I don't think this is inconsistent with the desire that you say was
expressed in Quebec for it to be "as good and complete as possible".
Discarding these sections would only make it less complete, without
actually making it better in any way.

But, that said ...

> and the focus shifted towards a more detailed section 4).

... I agree there isn't a lot to be gained by bikeshedding over these
old tests now, unless there are serious concerns about the validity
of the process they employed, that would make them bad examples to
follow, for people who wished to do later testing.

The focus of future editing should mainly be on the relevance and
validity of any new tests that people wish to add to it as they come.

I completely agree that it's highly desirable for us to avoid misleading
"the industry" about the characteristics of the final RFC defined codec,
so results should only be added there after WG scrutiny and consensus.

Cheers,
Ron