Return-Path: X-Original-To: codec@ietfc.amsl.com Delivered-To: codec@ietfc.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id E9A0AE074B for ; Tue, 19 Apr 2011 06:51:51 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.808 X-Spam-Level: X-Spam-Status: No, score=-2.808 tagged_above=-999 required=5 tests=[AWL=-0.210, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id b9AjrFejOYDS for ; Tue, 19 Apr 2011 06:51:50 -0700 (PDT) Received: from mail-vx0-f172.google.com (mail-vx0-f172.google.com [209.85.220.172]) by ietfc.amsl.com (Postfix) with ESMTP id 855B2E06F8 for ; Tue, 19 Apr 2011 06:51:50 -0700 (PDT) Received: by vxg33 with SMTP id 33so5383047vxg.31 for ; Tue, 19 Apr 2011 06:51:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=E2/8wbFCzYyq42dG2RgfL0WLquFc/hme8Kz9Oei6sUE=; b=NvXhwt5qpxJCkl9L+2/vA1qMckgrtnsm6cZyVMfshMfOUkV9L0zq+h9kb/xzHC1AhV S9nzdsTIcv7U3N2J5X6d1uF+fYlusxiNpsh36jIMHsCMfC+VDb5ra5lA4K6oJ698jBwj lrIZc9CV85G4Njq5QeLYik3uLZS8jbJ615jyw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=pBKIXm/X72siTQCuZBYP2qR3XG5dPdkNIMfXm1FN7CDeribkf8VqvZHxPshjGPkExc Dse7gjJKVzaLbxrVZvWu8HbPjeRt50wunqC9Y2bBrqMvI2YZCtho1tK+F68lnn5aumXq YnISYausVNQo7QWb4S5n+7p72Dh8K2q0YFe0E= MIME-Version: 1.0 Received: by 10.52.0.136 with SMTP id 8mr726667vde.45.1303221110001; Tue, 19 Apr 2011 06:51:50 -0700 (PDT) Received: by 10.52.168.6 with HTTP; Tue, 19 Apr 2011 06:51:49 -0700 (PDT) In-Reply-To: <20110419124432.GG31013@audi.shelbyville.oz> References: <20110419124432.GG31013@audi.shelbyville.oz> Date: Tue, 19 Apr 2011 09:51:49 -0400 Message-ID: From: Stephen Botzko To: Ron Content-Type: multipart/alternative; boundary=20cf304346f4534fa704a145cffd Cc: codec@ietf.org Subject: Re: [codec] Testing: A Novel Proposal X-BeenThere: codec@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Codec WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Apr 2011 13:51:52 -0000 --20cf304346f4534fa704a145cffd Content-Type: text/plain; charset=ISO-8859-1 I don't think this testing process is intended to be adversarial, it certainly is not for me. Also, I think the main goal is to benchmark the final codec's performance, not so much to drive improvements. Regards, Stephen Botzko On Tue, Apr 19, 2011 at 8:44 AM, Ron wrote: > > Hi Anisse, > > On Tue, Apr 19, 2011 at 10:38:28AM +0000, Anisse Taleb wrote: > > If you refer to the draft proposal, we are in a presence of a multimode > > multi-bandwidth codec, the per-bandwidth tests proposed are small in > > comparison to other tests for other codecs, however putting all these > > together makes the test quite large. For instance, ITU-T codecs usually > > address a single bandwidth. > > I don't think there's really all that much confusion as to how you arrived > at the particular initial combination of tests that you did. The important > part now though is determining what key things you hoped to learn from them > and figuring out how you might distill that into something tractable, which > somebody might actually perform and present to the group. > > Perhaps you could pick some representative tests out of that set, and use > those to perform some initial trials, to guide selection of a second round > of tests? > > > It was never the intention to make this an impossible test/task neither > was > > the intention to delay the publication of this codec. I am disappointed > that > > some of the codec proponents and supporters are defensive and quite > > dismissive. This is an opportunity for everyone to show and demonstrate > the > > quality of the codec in a hopefully well designed test that could be > > reproduced at will and hence not be easily dismissed or argued against. > > I don't think I've seen anyone on either side of any argument here, ever > be dismissive of testing. And the "proponents" that I've spoken to are > actually, in all good faith not at all feeling defensive about this codec. > > We all really, really, do want you to throw the absolute worst you can > conceive of at it. Because that's exactly how it got to be as good as > it is already. > > So please, please, please, please, please. With a new beach-car on top. > Please just do whatever test you think is the most significant that has > so far been omitted. > > That's the one we all really want to see the results of. But we can't > tell you what it should be, or argue with you over what it shouldn't. > We're all happy to give you what advice we can. But at the end of the > day, you have to decide, you or your peers need to conduct it, and then > we all need to see what it is that you've really taught us. > > That's how engineering works, right? > > > > As I repeatedly said this was an initial proposal and I value the > technical > > comments and feedback by Jean-Marc, Roman, Stephen, Koen and others in > and > > outside this mailing list. I am working on an update of that test plan > and is > > hopeful to send a version soon. Please let me know if you have any > comments > > such that I can accommodate your point of view too, even if you have > little > > horns :-). > > Some of us may keep ours sharp -- but I assure you we really all are > eagerly awaiting the results of the best test you can perform. And > have no sane reason to wish you anything but well in that task. > > > When it comes to your proposal, "If it isn't good enough... Prove it." > > > > I do not think I have anything to say about that except perhaps to remind > you > > of your own statements 2 years ago, in response to Roni, > > > > "a lot of characterization, testing, and documentation needs to be done. > > We're aware of that and we have a lot of work to do." > > And I remember somebody saying that G.711 should be good enough for anyone. > > There's been two years of characterisation, testing, and documentation done > to show that was utter bunkum. We really want you to show us what we've > missed. It's our turn now, not to believe *you* can do it, until you do! > :) > > But really, I do mean that in the spirit of really hoping you do find some > thing that lets us make it better. This is why we get this stuff reviewed > by experts - we really do want to see if your best game is better than > ours. > > Doesn't that sound like fun to you? > > Respectfully, > Ron > > > _______________________________________________ > codec mailing list > codec@ietf.org > https://www.ietf.org/mailman/listinfo/codec > --20cf304346f4534fa704a145cffd Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I don't think this testing process is intended to be adversarial, it ce= rtainly is not for me.

Also, I think the main goal is to benchmark t= he final codec's performance, not so much to drive improvements.

Regards,
Stephen Botzko


On Tue= , Apr 19, 2011 at 8:44 AM, Ron <ron@debian.org> wrote:

Hi Anisse,

On Tue, Apr 19, 2011 at 10:38:28AM +0000, Anisse Taleb wrote:
> If you refer to the draft proposal, we are in a presence of a multimod= e
> multi-bandwidth codec, the per-bandwidth tests proposed are small in > comparison to other tests for other codecs, however putting all these<= br> > together makes the test quite large. For instance, ITU-T codecs usuall= y
> address a single bandwidth.

I don't think there's really all that much confusion as to ho= w you arrived
at the particular initial combination of tests that you did. =A0The importa= nt
part now though is determining what key things you hoped to learn from them=
and figuring out how you might distill that into something tractable, which=
somebody might actually perform and present to the group.

Perhaps you could pick some representative tests out of that set, and use those to perform some initial trials, to guide selection of a second round<= br> of tests?

> It was never the intention to make this an impossible test/task neithe= r was
> the intention to delay the publication of this codec. I am disappointe= d that
> some of the codec proponents and supporters are defensive and quite > dismissive. This is an opportunity for everyone to show and demonstrat= e the
> quality of the codec in a hopefully well designed test that could be > reproduced at will and hence not be easily dismissed or argued against= .

I don't think I've seen anyone on either side of any argument= here, ever
be dismissive of testing. =A0And the "proponents" that I've s= poken to are
actually, in all good faith not at all feeling defensive about this codec.<= br>
We all really, really, do want you to throw the absolute worst you can
conceive of at it. =A0Because that's exactly how it got to be as good a= s
it is already.

So please, please, please, please, please. =A0With a new beach-car on top.<= br> Please just do whatever test you think is the most significant that has
so far been omitted.

That's the one we all really want to see the results of. =A0But we can&= #39;t
tell you what it should be, or argue with you over what it shouldn't. We're all happy to give you what advice we can. =A0But at the end of th= e
day, you have to decide, you or your peers need to conduct it, and then
we all need to see what it is that you've really taught us.

That's how engineering works, right?


> As I repeatedly said this was an initial proposal and I value the tech= nical
> comments and feedback by Jean-Marc, Roman, Stephen, Koen and others in= and
> outside this mailing list. I am working on an update of that test plan= and is
> hopeful to send a version soon. Please let me know if you have any com= ments
> such that I can accommodate your point of view too, even if you have l= ittle
> horns :-).

Some of us may keep ours sharp -- but I assure you we really all are<= br> eagerly awaiting the results of the best test you can perform. =A0And
have no sane reason to wish you anything but well in that task.

> When it comes to your proposal, "If it isn't good enough... P= rove it."
>
> I do not think I have anything to say about that except perhaps to rem= ind you
> of your own statements 2 years ago, in response to Roni,
>
> "a lot of characterization, testing, and documentation needs to b= e done.
> We're aware of that and we have a lot of work to do."

And I remember somebody saying that G.711 should be good enough for a= nyone.

There's been two years of characterisation, testing, and documentation = done
to show that was utter bunkum. =A0We really want you to show us what we'= ;ve
missed. =A0It's our turn now, not to believe *you* can do it, until you= do! :)

But really, I do mean that in the spirit of really hoping you do find some<= br> thing that lets us make it better. =A0This is why we get this stuff reviewe= d
by experts - we really do want to see if your best game is better than ours= .

Doesn't that sound like fun to you?

Respectfully,
Ron


_______________________________________________
codec mailing list
codec@ietf.org
h= ttps://www.ietf.org/mailman/listinfo/codec

--20cf304346f4534fa704a145cffd--