Re: [codec] A concrete proposal for requirements and testing

[Ron <[e-mail]>]

On Fri, Apr 08, 2011 at 11:32:33PM +0300, Roni Even wrote:
> And as stated in the charter on deliverables:
> 
> " Specification of a codec that meets the agreed-upon requirements"
> 
> I think there are no agreed-upon requirements yet and this is what I see as
> the  current discussion and not if the codec has fulfilled the requirements.

Thanks for the change of tack.  Since it seems unlikely that the WG
document draft-ietf-codec-requirements-02 would be amended to impose
a quality requirement of "Better than transparent", this current
thread has clearly been sailing a bit too close to the wind so far.

Which parts of that document do you still dispute or think need
improving?

That seems like precisely the sort of thing we should be finalising
now, rather than embarking upon arduous additional quality testing
that all indications are will only tell us things we already know.


> From: [e-mail] [mailto:[e-mail]] On Behalf Of Paul Coverdale
> Sent: Friday, April 08, 2011 9:32 PM
> Koen Vos wrote: 
> > quality has been shown to be good enough for the codec to be useful...
> 
> I think that’s where some people have difficulty. There’s been no systematic
> attempt to evaluate Opus against the performance requirements given in the
> codec requirements document (as thin as it is) in a controlled and repeatable
> manner.

I think that's a bit disingenuous to the careful methodology that was
employed by the developers, and to the people who so far have done
blind and scientific tests of their results, but without dwelling on
that slur:

There's also been no, even informal, results presented to suggest that
it does not exceed even the most ambitious performance requirement
expectations held at the outset, by a rather surprising margin.

There has certainly been a systematic attempt to cast doubt upon the
results that we do have, but so far not a shred of even questionable
evidence to back those doubts up.

Do you have some results to share, that back up the claims of the
people "having difficulty" accepting what has been achieved to date?


While I respect your adherence to processes that are indeed necessary
for the ITU to licence a technology and deploy it in the telephone
network, one of the very reasons for forming this group under the
auspices of the IETF was that the reality of developing and deploying
internet services is quite different.  We don't have just a handful
of companies responsible for signing off on a spec, and making do if
it later proves insufficient - we have millions of them, most of which
haven't even heard of this group yet.  And the best way to engage them
is to provide a proposed standard which they can work to and assess
for their own uses.

If we'd had to wait for DNS and IPv6 to mature before continuing that
process, our jaws wouldn't be dropping at what this group has produced
for its first proposed release, like they are today.

We have some very clever and very experienced people here.  If none of
them can present us with actual engineering faults (as opposed to minor
procedural details), then it's time to put our work to the real test.

Every day that goes by in which such faults aren't found and presented
is further evidence to support the appropriateness of the Last Call.


Cheers,
Ron