Re: [codec] draft test and processing plan for the IETF Codec

[Stephen Botzko <[e-mail]>]

We are not yet at WG LC, and the RFC publication process itself takes many
months.  So I am not seeing the scheduling issues as insurmountable. We
could easily require some core tests prior to WG LC, and continue testing
during the publication process.

I think the best course right now is to refine the test plan, and sort out
the process steps/scheduling when we have stronger consensus on the test
plan itself.

I agree that the tests need to be run on the version we plan to submit for
publication.  Making improvements later on does not have to mean the entire
test suite needs to be re-run - subsequent testing could be targeted to test
the changes.

Stephen Botzko

On Wed, Apr 13, 2011 at 9:38 AM, Benjamin M. Schwartz <
[e-mail]> wrote:

> On 04/13/2011 03:32 AM, Anisse Taleb wrote:
> > Please find attached a first draft of a test plan of the IETF codec
> (Opus).
>
> Thank you for drawing up this test plan, which clearly required a great
> deal of thought.  The results of such testing would certainly be very
> interesting to many.
>
> However, I think the execution of such a test is clearly _not_ an
> appropriate prerequisite for publishing a Proposed Standard.  By my
> calculations, the draft plan presently calls for over 1300 hours of
> listening tests, counting only audio being played, estimating 10-second
> samples and the minimum number of listeners.  Even if many listeners are
> listening in parallel, and overheads (such as delays between samples) are
> low, conducting such a test would still take many months.
>
> Such an extensive, expensive battery of tests can hardly be justified on
> some arbitrary codec version still under development.  It can only be
> justified if the codec being tested is not going to change, so that the
> sponsoring organizations can use the results to determine whether the
> codec meets their performance goals.
>
> Let's standardize, and then invite ultra-comprehensive systematic
> characterization.
>
> --Ben
>
>
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec
>
>