Re: [codec] Audio tests: Further steps

[Paul Coverdale <[e-mail]>]

I don't know why you're pouring scorn on this exercise, Ron. It seems to me
that it is a bona-fide attempt to understand the strengths and weaknesses of
the Opus codec in a controlled, unbiased manner, what a characterisation
test should do. It should have been done as part of the IETF codec WG
activity, but better late than never.

Cheers,

...Paul

>-----Original Message-----
>From: [e-mail] [mailto:[e-mail]] On Behalf
>Of Ron
>Sent: Tuesday, April 23, 2013 3:31 PM
>To: [e-mail]
>Cc: [e-mail]
>Subject: Re: [codec] Audio tests: Further steps
>
>On Tue, Apr 23, 2013 at 09:50:16AM +0200, Christian Hoene wrote:
>> Hi,
>>
>> currently, the codec comparison tests are running. Because of the
>> request of many codec developers, we plan to extend those tests: We
>> might add audio tests in which the content is varied to a large
>> extend. For that, we need sample that cannot be compressed well by
>> Opus or AAC-eLD. For me, it is easy to get those difficult samples for
>> Opus. It is much challenging to get those for AAC-eLD. Thus, if
>> somebody had to time to study the weaknesses of AAC-eLD, please
>forward me the samples.
>
>Uhm, so ...  while I'm certain that the codec developers will be
>delighted if you can point out any new killer samples that they aren't
>yet aware of (since significant work has already been made to improve
>the encoder for the known ones, and that work is still ongoing) -- I'm
>also pretty certain that going out of your way to deliberately select
>such samples immediately disqualifies this from being characterised as a
>"comparison test", or at least claiming that it's even remotely
>representative of what people will observe over a general corpus of
>their own audio, given the degree to which such samples really are
>outliers.
>
>> I cannot start fair tests if I do not have challenging samples for
>> both codecs.
>
>While such a test might have some novelty value to show "here are some
>non-exhaustive results for the worst samples that we could find in a few
>days of searching", I'm pretty sure words like "fair" and "scientific
>rigour" don't really belong in the same sentence.  Not in the least when
>you also say "we have the established list for one codec, but the known
>killers for the other is at present entirely unknown to us".
>
>If you want to spend your time doing that, that's fine, and the results
>may well be 'interesting'.  But mischaracterising them as a "comparison"
>test would just be somewhere on the spectrum from "mildly amusing" to "a
>sad day for Modern Science".
>
>It's your reputation though, and I can't tell you how to spend it.
>But you might want to think this through a little better if you are
>going to paint this with the brush of Being Science.
>
>This isn't the cosmetics industry, other people can measure these things
>too, and will continue to for some time to come.
>
> Cheers,
> Ron
>
>
>_______________________________________________
>codec mailing list
>[e-mail]
>https://www.ietf.org/mailman/listinfo/codec