Re: [codec] Audio tests: Further steps

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/23/2013 05:34 PM, Paul Coverdale wrote:
> I don't know why you're pouring scorn on this exercise, Ron. It
> seems to me that it is a bona-fide attempt to understand the
> strengths and weaknesses of the Opus codec in a controlled,
> unbiased manner, what a characterisation test should do. It should
> have been done as part of the IETF codec WG activity, but better
> late than never.

It's indeed a way to see what the strengths and weaknesses of a codec
are. I think what Ron mostly meant is that any *average* you compute
on such test would not be representative of which codec is better than
the other. Essentially, carefully picking out-liars is the worst form
of sampling you can have. It's useful for developers (knowing what to
focus on assuming you don't already know), but not for making general
quality conclusions.

	Jean-Marc

> Cheers,
> 
> ...Paul
> 
>> -----Original Message----- From: codec-bounces@ietf.org
>> [mailto:codec-bounces@ietf.org] On Behalf Of Ron Sent: Tuesday,
>> April 23, 2013 3:31 PM To: codec@ietf.org Cc:
>> cs.wg2.qualinet@listes.epfl.ch Subject: Re: [codec] Audio tests:
>> Further steps
>> 
>> On Tue, Apr 23, 2013 at 09:50:16AM +0200, Christian Hoene wrote:
>>> Hi,
>>> 
>>> currently, the codec comparison tests are running. Because of
>>> the request of many codec developers, we plan to extend those
>>> tests: We might add audio tests in which the content is varied
>>> to a large extend. For that, we need sample that cannot be
>>> compressed well by Opus or AAC-eLD. For me, it is easy to get
>>> those difficult samples for Opus. It is much challenging to get
>>> those for AAC-eLD. Thus, if somebody had to time to study the
>>> weaknesses of AAC-eLD, please
>> forward me the samples.
>> 
>> Uhm, so ...  while I'm certain that the codec developers will be 
>> delighted if you can point out any new killer samples that they
>> aren't yet aware of (since significant work has already been made
>> to improve the encoder for the known ones, and that work is still
>> ongoing) -- I'm also pretty certain that going out of your way to
>> deliberately select such samples immediately disqualifies this
>> from being characterised as a "comparison test", or at least
>> claiming that it's even remotely representative of what people
>> will observe over a general corpus of their own audio, given the
>> degree to which such samples really are outliers.
>> 
>>> I cannot start fair tests if I do not have challenging samples
>>> for both codecs.
>> 
>> While such a test might have some novelty value to show "here are
>> some non-exhaustive results for the worst samples that we could
>> find in a few days of searching", I'm pretty sure words like
>> "fair" and "scientific rigour" don't really belong in the same
>> sentence.  Not in the least when you also say "we have the
>> established list for one codec, but the known killers for the
>> other is at present entirely unknown to us".
>> 
>> If you want to spend your time doing that, that's fine, and the
>> results may well be 'interesting'.  But mischaracterising them as
>> a "comparison" test would just be somewhere on the spectrum from
>> "mildly amusing" to "a sad day for Modern Science".
>> 
>> It's your reputation though, and I can't tell you how to spend
>> it. But you might want to think this through a little better if
>> you are going to paint this with the brush of Being Science.
>> 
>> This isn't the cosmetics industry, other people can measure these
>> things too, and will continue to for some time to come.
>> 
>> Cheers, Ron
>> 
>> 
>> _______________________________________________ codec mailing
>> list codec@ietf.org https://www.ietf.org/mailman/listinfo/codec
> 
> _______________________________________________ codec mailing list 
> codec@ietf.org https://www.ietf.org/mailman/listinfo/codec
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRdwFPAAoJEJ6/8sItn9q9rJ0H+wWa8ljN6O8sPbbE/KetI8/G
c0qPuWWHdm3fACOm5iJMYDPtUmYS5vIMjLNZTP4/e1yavCNm1PBTek4vAMsIzd4h
h6yXCP2OymhawxJlkltoHGEg0BzpQ6IN2Io3VBpgISKzy74y7h9sI5vRAwY3Wpl9
bgO1PO4E+XlyZlHPAEqp5qugNN1fLeCRWsXNJynh3JKwxuyhWOO2cSHHyI8vsVzf
N8jG/YBgpdPVJ8/lqyYvaJhJ/grUUFxb9n2ECVovCbHWVGfdrHCUpeo7JxdIsWkY
AqUDtWKwgRrx2DQVD8TYdYcoeqPgosrzi2ya7yXK98XrZizIESIaGFv82STSVkM=
=4NdC
-----END PGP SIGNATURE-----