Re: [codec] A concrete proposal for requirements and testing

[Paul Coverdale <[e-mail]>]

>> > quality has been shown to be good enough for the codec to be
>useful...
>>
>> I think that’s where some people have difficulty. There’s been no
>systematic
>> attempt to evaluate Opus against the performance requirements given in
>the
>> codec requirements document (as thin as it is) in a controlled and
>repeatable
>> manner.
>
>I think that's a bit disingenuous to the careful methodology that was
>employed by the developers, and to the people who so far have done
>blind and scientific tests of their results, but without dwelling on
>that slur:

I'm not trying to be disingenuous. I just haven't seen details of the "careful methodology that was employed". There may well be a lot of results floating around behind the scenes, but the only official information that I have seen so far is the slide deck "Opus Testing" presented during IETF80. This did not provide a lot of experimental detail, and the number of subjects was quite small. Anyone familiar with subjective testing knows that the results can be heavily influenced by the methodology employed.

>
>There's also been no, even informal, results presented to suggest that
>it does not exceed even the most ambitious performance requirement
>expectations held at the outset, by a rather surprising margin.

We need a test plan before we can conduct any useful tests.


>
>Do you have some results to share, that back up the claims of the
>people "having difficulty" accepting what has been achieved to date?
>

Many people have indicated they will be willing to conduct tests, and share results, when a test plan is available.


>
>While I respect your adherence to processes that are indeed necessary
>for the ITU to licence a technology and deploy it in the telephone
>network, one of the very reasons for forming this group under the
>auspices of the IETF was that the reality of developing and deploying
>internet services is quite different.  We don't have just a handful
>of companies responsible for signing off on a spec, and making do if
>it later proves insufficient - we have millions of them, most of which
>haven't even heard of this group yet.  And the best way to engage them
>is to provide a proposed standard which they can work to and assess
>for their own uses.
>

ITU doesn't license technologies, that's a matter between individual organizations who may want to implement a specific standard which incorporates IPR. But in any case, the processes they follow are not there because of licensing. They are there to provide a logical progression from determining the requirements, developing solution(s), and testing the solution(s) against those requirements. 


Regards,

...Paul