Re: [rtcweb] Confirmation of consensus on audio codecs

[Stefan Hakansson LK <[e-mail]>]

On 08/29/2012 03:10 PM, Jean-Marc Valin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 08/29/2012 05:51 AM, Stefan Hakansson LK wrote:
...
>
>> That is great, but sort of underlines that there would be no harm
>> in delaying the decision until there are experiences made from real
>> world use - 'cause it would not be that long till that experience
>> has been made (Markus also brought up the IPR status as a reason
>> for waiting - I have no idea how long it would take to know more
>> about that).
>
> As Harald is pointing out, rtcweb implementations are going to ship
> pretty soon.

If that is the case (and I think and hope so), why would we need to make 
it MTI before seeing it in action?

[...]

> Are you expecting *another* single, standardized,
> royalty-free codec that operates over vast ranges of bitrates and
> operating conditions, from narrowband speech to stereo music, all with
> low delay, coming out in the next year? If not, what are you really
> waiting for?

No, I'm not expecting that. I would just prefer us to see that Opus does 
indeed deliver as promised before making it MTI. If it does, fine. If 
not, we'd have to discuss what to do then.

To me it is like if you're going to place a bet, for an upcoming big 
race, on a horse that has never been in an actual race, but shows great 
promise. If you know that the horse is going to participate in a few 
practice races soon, would you not prefer to wait and see how it fares 
in those before placing the bet (given that the odds would not change)?

Anyway, that's my view. Let's see what the chairs say.
>
>> As Paul suggested, I was referring to the lack of formal,
>> controlled, characterization tests. That is how other SDOs do it. I
>> don't think that is the only way to do it, but I think we should at
>> least have either such tests conducted or experience from
>> deployment and use (in a wide range of conditions and device types)
>> before making it MTI.
>
> Opus has had "ITU-style" testing on English (
> http://www.opus-codec.org/comparison/GoogleTest1.pdf ) and Mandarin (
> http://www.opus-codec.org/comparison/GoogleTest2.pdf ). And if you
> don't trust Google on the tests I linked to, it's also been tested by
> Nokia:
> http://research.nokia.com/files/public/%5B16%5D_InterSpeech2011_Voice_Quality_Characterization_of_IETF_Opus_Codec.pdf
Come on, those tests are very limited compared to a formal 
characterization test. (Example: 
www.itu.int/dms_pub/itu-t/opb/tut/T-TUT-ASC-2010-MSW-E.docx). There is 
very little info on the material used, the environment, processing and 
scripts and so on. And there seems to be no tests whatsoever (at least 
not involving humans) with actual channels introducing jitter and losses.

Note: I am in no way proposing that a formal characterization test is 
needed, or even the right thing to do. It is a costly and time consuming 
process, and alternative approaches could prove to be more efficient. 
What I am saying is that I think we should not mandate a codec that has 
neither gone through that kind of formal characterization testing nor 
has any field experience from actual use on a reasonable scale (covering 
different conditions and device types). It just seems wrong, especially 
given that we will soon have field experience.

>
> Now, unlike other SDOs, the testing did not stop there. ITU-T codecs
> generally end up being testing with something in the order of tens of
> minutes worth of audio. In *addition* to that kind of testing, Opus
> also had automated testing with hundreds of years worth of audio. If
> anything, I think other SDOs should learn something here.

This may very well be true. I guess this comes down to how much you 
trust that the quality assessment models (e.g. PEAQ, POLQA) give the 
same result as human test subjects would. But I think this sounds like a 
really good thing.

>
> 	Jean-Marc
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJQPhThAAoJEJ6/8sItn9q9nAcH/0BsXl/FBgMhbYWVALpNmIYi
> Yj1ezyW8JVSv4io1YIozucl6KiDVi3LwYFteYgl78PlQ7o2muv/lJP9VswRHooH5
> Gax+aJUZkz7SlKxn+25wUlLgKQw6GAcUID5sJ0ewWdGOrwZu+SlZgYF3egYmmwyL
> pEl8BFj7wu3uakRm/BE9LsUDPFdCoZS6uXqXRJOy9P1dZ25edvgaymQvOwhEcXu9
> xMq82YPNU2CZpfb88ew/l+U9FjW8c4iGkZYiu+VzmBL8wYOFP5pBhBhWc/bEb0WX
> 8IUC5PJ0MMp49kb7/kfQj65/Py7u6ytYaO4PA0rjMHJ2V1PzH4EuqtNweYXVzNI=
> =nwlD
> -----END PGP SIGNATURE-----
>