Re: [codec] Audio tests: Further steps

[Ron <[e-mail]>]

On Tue, Apr 23, 2013 at 09:50:16AM +0200, Christian Hoene wrote:
> Hi,
> 
> currently, the codec comparison tests are running. Because of the request of
> many codec developers, we plan to extend those tests: We might add audio
> tests in which the content is varied to a large extend. For that, we need
> sample that cannot be compressed well by Opus or AAC-eLD. For me, it is easy
> to get those difficult samples for Opus. It is much challenging to get those
> for AAC-eLD. Thus, if somebody had to time to study the weaknesses of
> AAC-eLD, please forward me the samples.

Uhm, so ...  while I'm certain that the codec developers will be delighted
if you can point out any new killer samples that they aren't yet aware of
(since significant work has already been made to improve the encoder for
the known ones, and that work is still ongoing) -- I'm also pretty certain
that going out of your way to deliberately select such samples immediately
disqualifies this from being characterised as a "comparison test", or at
least claiming that it's even remotely representative of what people will
observe over a general corpus of their own audio, given the degree to which
such samples really are outliers.

> I cannot start fair tests if I do not have challenging samples for both
> codecs.

While such a test might have some novelty value to show "here are some
non-exhaustive results for the worst samples that we could find in a few
days of searching", I'm pretty sure words like "fair" and "scientific
rigour" don't really belong in the same sentence.  Not in the least when
you also say "we have the established list for one codec, but the known
killers for the other is at present entirely unknown to us".

If you want to spend your time doing that, that's fine, and the results
may well be 'interesting'.  But mischaracterising them as a "comparison"
test would just be somewhere on the spectrum from "mildly amusing" to
"a sad day for Modern Science".

It's your reputation though, and I can't tell you how to spend it.
But you might want to think this through a little better if you are
going to paint this with the brush of Being Science.

This isn't the cosmetics industry, other people can measure these things
too, and will continue to for some time to come.

 Cheers,
 Ron