Re: [rtcweb] Confirmation of consensus on audio codecs

Stefan Hakansson LK <stefan.lk.hakansson@ericsson.com> Thu, 30 August 2012 11:58 UTC

Return-Path: <stefan.lk.hakansson@ericsson.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D0DD521F859C for <rtcweb@ietfa.amsl.com>; Thu, 30 Aug 2012 04:58:18 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.076
X-Spam-Level:
X-Spam-Status: No, score=-6.076 tagged_above=-999 required=5 tests=[AWL=0.173, BAYES_00=-2.599, HELO_EQ_SE=0.35, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CkTpCEBCKOCK for <rtcweb@ietfa.amsl.com>; Thu, 30 Aug 2012 04:58:17 -0700 (PDT)
Received: from mailgw2.ericsson.se (mailgw2.ericsson.se [193.180.251.37]) by ietfa.amsl.com (Postfix) with ESMTP id C797021F859E for <rtcweb@ietf.org>; Thu, 30 Aug 2012 04:58:16 -0700 (PDT)
X-AuditID: c1b4fb25-b7f046d00000644c-f6-503f55566166
Received: from esessmw0237.eemea.ericsson.se (Unknown_Domain [153.88.253.125]) by mailgw2.ericsson.se (Symantec Mail Security) with SMTP id 84.60.25676.6555F305; Thu, 30 Aug 2012 13:58:14 +0200 (CEST)
Received: from [127.0.0.1] (153.88.115.8) by esessmw0237.eemea.ericsson.se (153.88.115.91) with Microsoft SMTP Server id 8.3.264.1; Thu, 30 Aug 2012 13:58:13 +0200
Message-ID: <503F5555.5040303@ericsson.com>
Date: Thu, 30 Aug 2012 13:58:13 +0200
From: Stefan Hakansson LK <stefan.lk.hakansson@ericsson.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120714 Thunderbird/14.0
MIME-Version: 1.0
To: Jean-Marc Valin <jmvalin@mozilla.com>
References: <9E2843EA-EBB9-40B3-898C-6B5216FAE7A5@cisco.com> <503CBB52.8070300@ericsson.com> <503CF224.5050609@mozilla.com> <503DE60D.2060706@ericsson.com> <503E14E1.6000401@mozilla.com>
In-Reply-To: <503E14E1.6000401@mozilla.com>
Content-Type: text/plain; charset="windows-1252"; format="flowed"
Content-Transfer-Encoding: 7bit
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFuplluLIzCtJLcpLzFFi42KZGfG3Vjcs1D7A4Px8TYv/Uzks1v5rZ3dg 8liy5CeTR9+BLtYApigum5TUnMyy1CJ9uwSujFcPrzAX3FOumLnoKksD4yTZLkZODgkBE4mp rV9ZIWwxiQv31rOB2EICpxglHvTkQ9jLGSXaLwuC2LwC2hJ3Jl1hB7FZBFQldh1ZzAJiswnY SKztnsIEYosKhEis+TaFEaJeUOLkzCdgNSICmhJ3fqwC62UWUJe4s/gcmC0sYC+x8cR/oHou oF27GCWmbG8CG8QJtOzQqWlACQ6gBnuJB1vLIHrlJba/ncMMcZuuxLvX91gnMArOQrJuFkLH LCQdCxiZVzEK5yZm5qSXG+mlFmUmFxfn5+kVp25iBAbpwS2/VXcw3jkncohRmoNFSZzXeuse fyGB9MSS1OzU1ILUovii0pzU4kOMTBycUg2MgvOTfjycmL4sZxHL2sknnHNFfKQCBRnll3/q z545m+PYfZFEzStiWbsbo+0M7tfcW/vb4Hnx04KjpVbrY603vbtVYnxvxezoM6dDzzgv2ct0 b2bLtn2Ve4LMrz89b/rMdLLljTi76pLM9eX3C0plwjJenM049OG+XU1eva3COS9G78cOZ8yT lViKMxINtZiLihMBBqH64CACAAA=
Cc: "rtcweb@ietf.org" <rtcweb@ietf.org>
Subject: Re: [rtcweb] Confirmation of consensus on audio codecs
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 30 Aug 2012 11:58:18 -0000

On 08/29/2012 03:10 PM, Jean-Marc Valin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 08/29/2012 05:51 AM, Stefan Hakansson LK wrote:
...
>
>> That is great, but sort of underlines that there would be no harm
>> in delaying the decision until there are experiences made from real
>> world use - 'cause it would not be that long till that experience
>> has been made (Markus also brought up the IPR status as a reason
>> for waiting - I have no idea how long it would take to know more
>> about that).
>
> As Harald is pointing out, rtcweb implementations are going to ship
> pretty soon.

If that is the case (and I think and hope so), why would we need to make 
it MTI before seeing it in action?

[...]

> Are you expecting *another* single, standardized,
> royalty-free codec that operates over vast ranges of bitrates and
> operating conditions, from narrowband speech to stereo music, all with
> low delay, coming out in the next year? If not, what are you really
> waiting for?

No, I'm not expecting that. I would just prefer us to see that Opus does 
indeed deliver as promised before making it MTI. If it does, fine. If 
not, we'd have to discuss what to do then.

To me it is like if you're going to place a bet, for an upcoming big 
race, on a horse that has never been in an actual race, but shows great 
promise. If you know that the horse is going to participate in a few 
practice races soon, would you not prefer to wait and see how it fares 
in those before placing the bet (given that the odds would not change)?

Anyway, that's my view. Let's see what the chairs say.
>
>> As Paul suggested, I was referring to the lack of formal,
>> controlled, characterization tests. That is how other SDOs do it. I
>> don't think that is the only way to do it, but I think we should at
>> least have either such tests conducted or experience from
>> deployment and use (in a wide range of conditions and device types)
>> before making it MTI.
>
> Opus has had "ITU-style" testing on English (
> http://www.opus-codec.org/comparison/GoogleTest1.pdf ) and Mandarin (
> http://www.opus-codec.org/comparison/GoogleTest2.pdf ). And if you
> don't trust Google on the tests I linked to, it's also been tested by
> Nokia:
> http://research.nokia.com/files/public/%5B16%5D_InterSpeech2011_Voice_Quality_Characterization_of_IETF_Opus_Codec.pdf
Come on, those tests are very limited compared to a formal 
characterization test. (Example: 
www.itu.int/dms_pub/itu-t/opb/tut/T-TUT-ASC-2010-MSW-E.docx). There is 
very little info on the material used, the environment, processing and 
scripts and so on. And there seems to be no tests whatsoever (at least 
not involving humans) with actual channels introducing jitter and losses.

Note: I am in no way proposing that a formal characterization test is 
needed, or even the right thing to do. It is a costly and time consuming 
process, and alternative approaches could prove to be more efficient. 
What I am saying is that I think we should not mandate a codec that has 
neither gone through that kind of formal characterization testing nor 
has any field experience from actual use on a reasonable scale (covering 
different conditions and device types). It just seems wrong, especially 
given that we will soon have field experience.

>
> Now, unlike other SDOs, the testing did not stop there. ITU-T codecs
> generally end up being testing with something in the order of tens of
> minutes worth of audio. In *addition* to that kind of testing, Opus
> also had automated testing with hundreds of years worth of audio. If
> anything, I think other SDOs should learn something here.

This may very well be true. I guess this comes down to how much you 
trust that the quality assessment models (e.g. PEAQ, POLQA) give the 
same result as human test subjects would. But I think this sounds like a 
really good thing.

>
> 	Jean-Marc
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJQPhThAAoJEJ6/8sItn9q9nAcH/0BsXl/FBgMhbYWVALpNmIYi
> Yj1ezyW8JVSv4io1YIozucl6KiDVi3LwYFteYgl78PlQ7o2muv/lJP9VswRHooH5
> Gax+aJUZkz7SlKxn+25wUlLgKQw6GAcUID5sJ0ewWdGOrwZu+SlZgYF3egYmmwyL
> pEl8BFj7wu3uakRm/BE9LsUDPFdCoZS6uXqXRJOy9P1dZ25edvgaymQvOwhEcXu9
> xMq82YPNU2CZpfb88ew/l+U9FjW8c4iGkZYiu+VzmBL8wYOFP5pBhBhWc/bEb0WX
> 8IUC5PJ0MMp49kb7/kfQj65/Py7u6ytYaO4PA0rjMHJ2V1PzH4EuqtNweYXVzNI=
> =nwlD
> -----END PGP SIGNATURE-----
>