Re: [codec] draft test and processing plan for the IETF Codec

Dear Jean Marc,

In any case the results would be there and we can analyze these in whatever form we want. However, requirements need to be set on the codec are you suggesting to defer the requirements discussion to after we see the results?

Kind regards,
/Anisse

> -----Original Message-----
> From: Jean-Marc Valin [mailto:jean-marc.valin@octasic.com]
> Sent: Tuesday, April 19, 2011 2:59 AM
> To: Anisse Taleb
> Cc: Paul Coverdale; codec@ietf.org
> Subject: Re: [codec] draft test and processing plan for the IETF Codec
> 
> On 11-04-18 07:37 PM, Anisse Taleb wrote:
> > JM, Greg, Paul, [taking emails in chronological order was ill advised
> > :-)]
> >
> > I do not disagree with the statistical pitfalls you mention. As Paul
> > stated and also what I wrote in a direct reply to this, there is no
> > single uber-requirement to be passed by the codec, rather a vector of
> > requirements that summarize the performance of the codec compared to
> > other codecs. These have to be analyzed and discussed one by one.
> 
> Then, I guess we have no need for BTs and NWTs in the test plan. In the
> end, once the results are analyzed, we'll be able to take each "codec
> pair" and say either "A is better than B", "B is better than A", or "A
> is tied with B" (null hypothesis). The WG members can then decide what
> to conclude from those.
> 
> Cheers,
> 
> 	Jean-Marc
> 
> > Kind regards, /Anisse
> >
> >> -----Original Message----- From: codec-bounces@ietf.org
> >> [mailto:codec-bounces@ietf.org] On Behalf Of Jean-Marc Valin Sent:
> >> Thursday, April 14, 2011 3:07 PM To: Paul Coverdale Cc:
> >> codec@ietf.org Subject: Re: [codec] draft test and processing plan
> >> for the IETF Codec
> >>
> >>> I don't think the situation is as dire as you make out. Your
> >>> analysis assumes that all requirements are completely
> >>> independent. This is not the case, in many cases if you meet one
> >>> requirement you are likely to meet others of the same kind (eg
> >>> performance as a function of bit rate).
> >>>
> >>> But in any case, the statistical analysis procedure outlined in
> >>> the test plan doesn't assume that every requirement must be met
> >>> with absolute certainty, it allows for a confidence interval.
> >>
> >> This is exactly what Greg is considering in his analysis. He's
> >> starting from the assumption that the codec really meets *all* 162
> >> requirements. Consider just the NWT requirements: if we were truly
> >> no worse than the reference codec, then with 87 tests against a 95%
> >> confidence interval, we would be expected to fail about 4 tests
> >> just by random chance. Considering both NWT and BT requirements,
> >> the odds of passing Anisse's proposed test plan given the
> >> assumptions above are 4.1483e-33. See http://xkcd.com/882/ for a
> >> more rigorous analysis.
> >>
> >> Cheers,
> >>
> >> Jean-Marc _______________________________________________ codec
> >> mailing list codec@ietf.org
> >> https://www.ietf.org/mailman/listinfo/codec
> > _______________________________________________ codec mailing list
> > codec@ietf.org https://www.ietf.org/mailman/listinfo/codec
> >
> >