Re: [codec] A concrete proposal for requirements and testing

[Kavan Seggie <[e-mail]>]

> We don't have just a handful
> of companies responsible for signing off on a spec, and making do if
> it later proves insufficient - we have millions of them, most of which
> haven't even heard of this group yet.  And the best way to engage them
> is to provide a proposed standard which they can work to and assess
> for their own uses.

+1

I am the Founder and CEO of a startup who has been quietly observing this WG
and the development of CELT for almost two years.

It would be great if we could progress to real world testing and get the
product out to the people. For us, SILK is an amazing codec, better than any
voice codec we have tried including proprietary technology from the best
vendors. CELT, sounds excellent too. We do not have the resources to run a
formal test plan so we trust the ears of our team, friends and family and we
think they are really solid.

IMHO the next step is to provide a release version of Opus to the wider
community for real world testing. They'll let you know if there are any real
issues far quicker than more formal testing.

Regards

Kavan


On Sat, Apr 9, 2011 at 4:06 AM, Ron <[e-mail]> wrote:

>
> On Fri, Apr 08, 2011 at 11:32:33PM +0300, Roni Even wrote:
> > And as stated in the charter on deliverables:
> >
> > " Specification of a codec that meets the agreed-upon requirements"
> >
> > I think there are no agreed-upon requirements yet and this is what I see
> as
> > the  current discussion and not if the codec has fulfilled the
> requirements.
>
> Thanks for the change of tack.  Since it seems unlikely that the WG
> document draft-ietf-codec-requirements-02 would be amended to impose
> a quality requirement of "Better than transparent", this current
> thread has clearly been sailing a bit too close to the wind so far.
>
> Which parts of that document do you still dispute or think need
> improving?
>
> That seems like precisely the sort of thing we should be finalising
> now, rather than embarking upon arduous additional quality testing
> that all indications are will only tell us things we already know.
>
>
> > From: [e-mail] [mailto:[e-mail]] On Behalf
> Of Paul Coverdale
> > Sent: Friday, April 08, 2011 9:32 PM
> > Koen Vos wrote:
> > > quality has been shown to be good enough for the codec to be useful...
> >
> > I think that’s where some people have difficulty. There’s been no
> systematic
> > attempt to evaluate Opus against the performance requirements given in
> the
> > codec requirements document (as thin as it is) in a controlled and
> repeatable
> > manner.
>
> I think that's a bit disingenuous to the careful methodology that was
> employed by the developers, and to the people who so far have done
> blind and scientific tests of their results, but without dwelling on
> that slur:
>
> There's also been no, even informal, results presented to suggest that
> it does not exceed even the most ambitious performance requirement
> expectations held at the outset, by a rather surprising margin.
>
> There has certainly been a systematic attempt to cast doubt upon the
> results that we do have, but so far not a shred of even questionable
> evidence to back those doubts up.
>
> Do you have some results to share, that back up the claims of the
> people "having difficulty" accepting what has been achieved to date?
>
>
> While I respect your adherence to processes that are indeed necessary
> for the ITU to licence a technology and deploy it in the telephone
> network, one of the very reasons for forming this group under the
> auspices of the IETF was that the reality of developing and deploying
> internet services is quite different.  We don't have just a handful
> of companies responsible for signing off on a spec, and making do if
> it later proves insufficient - we have millions of them, most of which
> haven't even heard of this group yet.  And the best way to engage them
> is to provide a proposed standard which they can work to and assess
> for their own uses.
>
> If we'd had to wait for DNS and IPv6 to mature before continuing that
> process, our jaws wouldn't be dropping at what this group has produced
> for its first proposed release, like they are today.
>
> We have some very clever and very experienced people here.  If none of
> them can present us with actual engineering faults (as opposed to minor
> procedural details), then it's time to put our work to the real test.
>
> Every day that goes by in which such faults aren't found and presented
> is further evidence to support the appropriateness of the Last Call.
>
>
> Cheers,
> Ron
>
>
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec
>



-- 
Kavan Seggie
SayMama <http://www.saymama.com> - The video chat social network.

Find us on Facebook: facebook.com/officialsaymama
Follow us on Twitter: twitter.com/saymama