Re: [codec] Testing: A Novel Proposal

[Ron <[e-mail]>]

Hi Anisse,

On Tue, Apr 19, 2011 at 10:38:28AM +0000, Anisse Taleb wrote:
> If you refer to the draft proposal, we are in a presence of a multimode
> multi-bandwidth codec, the per-bandwidth tests proposed are small in
> comparison to other tests for other codecs, however putting all these
> together makes the test quite large. For instance, ITU-T codecs usually
> address a single bandwidth.

I don't think there's really all that much confusion as to how you arrived
at the particular initial combination of tests that you did.  The important
part now though is determining what key things you hoped to learn from them
and figuring out how you might distill that into something tractable, which
somebody might actually perform and present to the group.

Perhaps you could pick some representative tests out of that set, and use
those to perform some initial trials, to guide selection of a second round
of tests?

> It was never the intention to make this an impossible test/task neither was
> the intention to delay the publication of this codec. I am disappointed that
> some of the codec proponents and supporters are defensive and quite
> dismissive. This is an opportunity for everyone to show and demonstrate the
> quality of the codec in a hopefully well designed test that could be
> reproduced at will and hence not be easily dismissed or argued against.

I don't think I've seen anyone on either side of any argument here, ever
be dismissive of testing.  And the "proponents" that I've spoken to are
actually, in all good faith not at all feeling defensive about this codec.

We all really, really, do want you to throw the absolute worst you can
conceive of at it.  Because that's exactly how it got to be as good as
it is already.

So please, please, please, please, please.  With a new beach-car on top.
Please just do whatever test you think is the most significant that has
so far been omitted.

That's the one we all really want to see the results of.  But we can't
tell you what it should be, or argue with you over what it shouldn't.
We're all happy to give you what advice we can.  But at the end of the
day, you have to decide, you or your peers need to conduct it, and then
we all need to see what it is that you've really taught us.

That's how engineering works, right?


> As I repeatedly said this was an initial proposal and I value the technical
> comments and feedback by Jean-Marc, Roman, Stephen, Koen and others in and
> outside this mailing list. I am working on an update of that test plan and is
> hopeful to send a version soon. Please let me know if you have any comments
> such that I can accommodate your point of view too, even if you have little
> horns :-).

Some of us may keep ours sharp -- but I assure you we really all are
eagerly awaiting the results of the best test you can perform.  And
have no sane reason to wish you anything but well in that task.

> When it comes to your proposal, "If it isn't good enough... Prove it."
> 
> I do not think I have anything to say about that except perhaps to remind you
> of your own statements 2 years ago, in response to Roni, 
> 
> "a lot of characterization, testing, and documentation needs to be done.
> We're aware of that and we have a lot of work to do."

And I remember somebody saying that G.711 should be good enough for anyone.

There's been two years of characterisation, testing, and documentation done
to show that was utter bunkum.  We really want you to show us what we've
missed.  It's our turn now, not to believe *you* can do it, until you do! :)

But really, I do mean that in the spirit of really hoping you do find some
thing that lets us make it better.  This is why we get this stuff reviewed
by experts - we really do want to see if your best game is better than ours.

Doesn't that sound like fun to you?

Respectfully,
Ron