Re: [codec] Summary of test results

Jean-Marc Valin <jean-marc.valin@octasic.com> Thu, 16 June 2011 21:07 UTC

Return-Path: <jean-marc.valin@octasic.com>
X-Original-To: codec@ietfa.amsl.com
Delivered-To: codec@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9D25511E82FD for <codec@ietfa.amsl.com>; Thu, 16 Jun 2011 14:07:57 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rvzjXRI9OxVs for <codec@ietfa.amsl.com>; Thu, 16 Jun 2011 14:07:57 -0700 (PDT)
Received: from toroondcbmts08-srv.bellnexxia.net (toroondcbmts08-srv.bellnexxia.net [207.236.237.42]) by ietfa.amsl.com (Postfix) with ESMTP id ECAFA11E8138 for <codec@ietf.org>; Thu, 16 Jun 2011 14:07:56 -0700 (PDT)
Received: from toip58-bus.srvr.bell.ca ([67.69.240.185]) by toroondcbmts08-srv.bellnexxia.net (InterMail vM.8.00.01.00 201-2244-105-20090324) with ESMTP id <20110616210756.BDDQ15007.toroondcbmts08-srv.bellnexxia.net@toip58-bus.srvr.bell.ca>; Thu, 16 Jun 2011 17:07:56 -0400
Received: from toip60-bus.srvr.bell.ca ([67.69.240.187]) by toip58-bus.srvr.bell.ca with ESMTP; 16 Jun 2011 17:07:47 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: Av0EAD1q+k3PPaAN/2dsb2JhbABSpmh3iHPAOIYnBJY1CYst
Received: from mail.octasic.com ([207.61.160.13]) by toip60-bus.srvr.bell.ca with ESMTP; 16 Jun 2011 17:07:47 -0400
Received: from [10.100.50.90] (10.100.50.90) by MAIL2.octasic.com (10.100.10.44) with Microsoft SMTP Server (TLS) id 14.1.270.1; Thu, 16 Jun 2011 17:07:47 -0400
Message-ID: <4DFA70A2.6030203@octasic.com>
Date: Thu, 16 Jun 2011 17:07:46 -0400
From: Jean-Marc Valin <jean-marc.valin@octasic.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: Christian Hoene <hoene@uni-tuebingen.de>
References: <4DFA5106.1040903@octasic.com> <003401cc2c63$18e75520$4ab5ff60$@uni-tuebingen.de>
In-Reply-To: <003401cc2c63$18e75520$4ab5ff60$@uni-tuebingen.de>
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit
X-Originating-IP: [10.100.50.90]
Cc: codec@ietf.org
Subject: Re: [codec] Summary of test results
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 16 Jun 2011 21:07:57 -0000

Hi Christian,

On 11-06-16 04:21 PM, Christian Hoene wrote:
> Thank you very much for your efforts.  Allow me to play again the role of
> the critic. For the readers of Opus characterization document it would be
> more useful to have the measured results in table (concrete figures).

Note that this draft is not meant to include all the details of the tests. 
For that, you should refer to the original results documents that are cited 
for each test. All of them are available on the net except for the most 
recent test: "Voice Quality Characterization of IETF Opus Codec" that will 
be presented by Anssi Ramo (from Nokia) at Interspeech 2011. However, he 
was kind enough to provide me with a copy and gave me permission to 
redistribute it to working group members upon request.

> Also, in order to plan further listening-only tests, an overview of missing
> test cases is missing.
>
> Do you plan to provide anything in this respect in a later update of the
> document? How is going to coordinate future tests? Or do been already enough
> tests been made?

As stated in the conclusion, we believe that the combination of all tests 
demonstrates that Opus is better than the codecs it needs to beat. Feel 
free to conduct any test you deem useful if you disagree with parts of the 
conclusions.

> Do you plan to have an additional informal IETF RFC with the
> characterization results? This would make sense but it requires changes in
> the charter...

At this point, this draft is not meant to ever become an RFC. It is only 
meant for discussion (friendlier than a lengthy post on the mailing list).

Cheers,

	Jean-Marc