Re: [codec] Testing: A Novel Proposal

[Stephen Botzko <[e-mail]>]

I don't think this testing process is intended to be adversarial, it
certainly is not for me.

Also, I think the main goal is to benchmark the final codec's performance,
not so much to drive improvements.

Regards,
Stephen Botzko


On Tue, Apr 19, 2011 at 8:44 AM, Ron <[e-mail]> wrote:

>
> Hi Anisse,
>
> On Tue, Apr 19, 2011 at 10:38:28AM +0000, Anisse Taleb wrote:
> > If you refer to the draft proposal, we are in a presence of a multimode
> > multi-bandwidth codec, the per-bandwidth tests proposed are small in
> > comparison to other tests for other codecs, however putting all these
> > together makes the test quite large. For instance, ITU-T codecs usually
> > address a single bandwidth.
>
> I don't think there's really all that much confusion as to how you arrived
> at the particular initial combination of tests that you did.  The important
> part now though is determining what key things you hoped to learn from them
> and figuring out how you might distill that into something tractable, which
> somebody might actually perform and present to the group.
>
> Perhaps you could pick some representative tests out of that set, and use
> those to perform some initial trials, to guide selection of a second round
> of tests?
>
> > It was never the intention to make this an impossible test/task neither
> was
> > the intention to delay the publication of this codec. I am disappointed
> that
> > some of the codec proponents and supporters are defensive and quite
> > dismissive. This is an opportunity for everyone to show and demonstrate
> the
> > quality of the codec in a hopefully well designed test that could be
> > reproduced at will and hence not be easily dismissed or argued against.
>
> I don't think I've seen anyone on either side of any argument here, ever
> be dismissive of testing.  And the "proponents" that I've spoken to are
> actually, in all good faith not at all feeling defensive about this codec.
>
> We all really, really, do want you to throw the absolute worst you can
> conceive of at it.  Because that's exactly how it got to be as good as
> it is already.
>
> So please, please, please, please, please.  With a new beach-car on top.
> Please just do whatever test you think is the most significant that has
> so far been omitted.
>
> That's the one we all really want to see the results of.  But we can't
> tell you what it should be, or argue with you over what it shouldn't.
> We're all happy to give you what advice we can.  But at the end of the
> day, you have to decide, you or your peers need to conduct it, and then
> we all need to see what it is that you've really taught us.
>
> That's how engineering works, right?
>
>
> > As I repeatedly said this was an initial proposal and I value the
> technical
> > comments and feedback by Jean-Marc, Roman, Stephen, Koen and others in
> and
> > outside this mailing list. I am working on an update of that test plan
> and is
> > hopeful to send a version soon. Please let me know if you have any
> comments
> > such that I can accommodate your point of view too, even if you have
> little
> > horns :-).
>
> Some of us may keep ours sharp -- but I assure you we really all are
> eagerly awaiting the results of the best test you can perform.  And
> have no sane reason to wish you anything but well in that task.
>
> > When it comes to your proposal, "If it isn't good enough... Prove it."
> >
> > I do not think I have anything to say about that except perhaps to remind
> you
> > of your own statements 2 years ago, in response to Roni,
> >
> > "a lot of characterization, testing, and documentation needs to be done.
> > We're aware of that and we have a lot of work to do."
>
> And I remember somebody saying that G.711 should be good enough for anyone.
>
> There's been two years of characterisation, testing, and documentation done
> to show that was utter bunkum.  We really want you to show us what we've
> missed.  It's our turn now, not to believe *you* can do it, until you do!
> :)
>
> But really, I do mean that in the spirit of really hoping you do find some
> thing that lets us make it better.  This is why we get this stuff reviewed
> by experts - we really do want to see if your best game is better than
> ours.
>
> Doesn't that sound like fun to you?
>
> Respectfully,
> Ron
>
>
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec
>