Re: [codec] A concrete proposal for requirements and testing

[Stephen Botzko <[e-mail]>]

in-line
Stephen Botzko

On Tue, Apr 12, 2011 at 1:19 AM, Ron <[e-mail]> wrote:

>
> Hi Anisse,
>
> On Mon, Apr 11, 2011 at 06:26:08PM +0000, Anisse Taleb wrote:
> > Hi all,
> >
> > I feel this thread has somewhat diverged from its initial purpose which
> was
> > to have a concrete plan for testing and requirements.
>
> Well, its initial purpose, as clearly seen in its first post, was to
> resolve
> the disagreement (or possibly confusion?) that resulted from trying to take
> a hum on those questions, without adequate discussion, at Prague.
>
> It diverged from that almost immediately, and has steadfastly refused to
> return to it.  If we can take that as meaning that nobody actually does
> disagree with the points that Jean-Marc asked to clarify, then I think
> we can take that as meaning this (original) thread has actually achieved
> its purpose and we now do have consensus on those points.
>

I don't see how anyone can think we've achieved consensus.  That is just
plain silly.  BTW, isn't up to *chairs* to call for consensus?

On the whole, the thread started reasonably, with good discussions on adding
some speech conditions, choosing the number of subjects, etc. The thread
diverged when AMR was proposed as an anchor in section 4.2. Jean-Marc
"strongly disagreed" to testing requirements against AMR-NB and AMR-WB,
saying it was "not fair", and suggesting that he thought Opus would fail in
that comparison (though he thought it would be close).  He suggested that
only unencumbered codecs be used as anchors for strict requirements. [He did
*not* object to the use of encumbered codecs in testing objectives, just in
testing "strict requirements"].

However, the idea of limiting IPR in the anchors was clearly not accepted by
others on the thread, and as is often the case on IETF lists, the mention of
IPR took us off-topic.  Then the idea of systematic testing itself became
the predominant hot topic, with many (like me) thinking it is essential, and
others (like yourself) thinking it is a waste of time.  Since the value of
systematic testing is a more basic discussion than the details of the test
plan, the thread never got back on track.

Starting over by contributing draft systematic test plan seems like the best
approach.


> Is there some formal IETF thing we need to do to acknowledge that now?
> (as separate from this subthread about an additional planned test)
>
> > I have also a feeling
> > that there are "cultural" differences and divergences as to what
> "testing"
> > means.
>
> I don't think anyone disagrees on what testing means.  I do think we have
> some disagreement on what tests are relevant though.  Some people think
> testing against G.711, or against codecs that cannot possibly fill the
> requirements this group set out to achieve, is a fairly pointless exercise.
> At least in the context of assessing whether we have met the requirements
> of the WG, and to what standard.  And in the face of the test results that
> we do already have.
>
> Such tests may still be interesting to see (I'd be interested in seeing
> them for one), but I strongly disagree that they are at all serving the
> aims of the WG, and don't think we should delay its progress if we don't
> have them in time.
>

I think you and Anisse have very different views on what it means to run a
proper test.


>
> > That aside, I would like to inform that a proposal for test plan is
> currently
> > being drafted and will be communicated this week.
>
> Thank you.  I'm very glad to hear that the people who weren't satisfied
> by the results we've already seen are taking the initiative to resolve
> that to their full satisfaction.  As only they can do.
>
> I look forward to seeing your plan, and its results.  As I have all of
> the other tests performed to date.
>
> Best,
> Ron
>
>
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec
>