Re: [codec] draft test and processing plan for the IETF Codec

Jean-Marc Valin <jmvalin@jmvalin.ca> Fri, 15 April 2011 04:50 UTC

Return-Path: <jmvalin@jmvalin.ca>
X-Original-To: codec@ietfc.amsl.com
Delivered-To: codec@ietfc.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id C0BB7E06AE for <codec@ietfc.amsl.com>; Thu, 14 Apr 2011 21:50:15 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ULmEEtYIpp7T for <codec@ietfc.amsl.com>; Thu, 14 Apr 2011 21:50:13 -0700 (PDT)
Received: from relais.videotron.ca (relais.videotron.ca [24.201.245.36]) by ietfc.amsl.com (Postfix) with ESMTP id 56BD8E0668 for <codec@ietf.org>; Thu, 14 Apr 2011 21:50:13 -0700 (PDT)
MIME-version: 1.0
Content-transfer-encoding: 7bit
Content-type: text/plain; charset="ISO-8859-1"; format="flowed"
Received: from [192.168.1.14] ([184.160.206.46]) by vl-mh-mrz25.ip.videotron.ca (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec 16 2008; 32bit)) with ESMTP id <0LJO004F0G109R70@vl-mh-mrz25.ip.videotron.ca> for codec@ietf.org; Fri, 15 Apr 2011 00:48:36 -0400 (EDT)
Message-id: <4DA7CE76.1040502@jmvalin.ca>
Date: Fri, 15 Apr 2011 00:49:58 -0400
From: Jean-Marc Valin <jmvalin@jmvalin.ca>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
To: Jean-Marc Valin <jmvalin@jmvalin.ca>
References: <F5AD4C2E5FBF304ABAE7394E9979AF7C26BC684E@LHREML503-MBX.china.huawei.com> <4DA7B88F.80002@jmvalin.ca>
In-reply-to: <4DA7B88F.80002@jmvalin.ca>
Cc: "codec@ietf.org" <codec@ietf.org>
Subject: Re: [codec] draft test and processing plan for the IETF Codec
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Apr 2011 04:50:15 -0000

So here's some more specific comments on actual bitrates:

1) For narrowband Speex, the rates currently listed are 8, 12, 16 kb/s. 
Those should be changed to 8, 11, 15 kb/s to match the actual Speex 
bitrates.

2) For iLBC, the rates currently listed are 8, 12, 16 kb/s. I think we 
should only use 15.2 kb/s for iLBC. There's another rate, which is 13.33 
kb/s but that's for 30 ms frames so it's not very interesting.

3) For Speex wideband, the rates currently listed are 12, 24, 32 kb/s. I 
think Speex wideband around 12 kb/s is just crap. Worth testing would be 
20.6 and 27.8 kb/s.

4) For super-wideband Speex, I recommend just dumping that. This Speex 
mode was a mistake right from the start and usually has worse quality 
than wideband Speex.

Regarding super-wideband, one thing to keep in mind is that Opus defines 
super-wideband as having a 12 kHz audio bandwidth (24 kHz sampling 
rate). This makes comparisons with other codecs more difficult. The 
rates currently listed for super-wideband are 24, 32, 64 kb/s. I 
recommend running 24 kb/s in super-wideband and running 32 and 64 kb/s 
in fullband mode (even if the input is a 32 kHz signal).

For the very low delay tests (10 ms frame size), I think all the listed 
rates should be using fullband mode except the 32 kb/s.

That's it for now. Any thoughts?

	Jean-Marc


On 11-04-14 11:16 PM, Jean-Marc Valin wrote:
> Hi Anisse,
>
> I gave some more thought on your proposed test plan and as Cullen
> suggested, I think the main cause of disagreement is not that much on
> the testing, but on the conditions for publishing (large number of BT,
> NWT). Considering that ultimately, the decision to publish a spec is
> always based on WG consensus, then I think that problem can be
> completely bypassed. Once we make it up to the individuals to decide,
> then we can focus on "simply" designing a good test.
>
> Overall I thought the conditions you were proposing in section 2 were
> pretty reasonable. There's a few details like selecting existing rates
> for codecs like Speex and iLBC, but that should be easy to solve. Once
> these are sorted out, interested parties (we had several hands raised in
> the last meeting) can start testing and we then let each individual
> decide on whether the codec is any good based on the results of the tests.
>
> Sounds like a plan?
>
> Jean-Marc
>
>
> On 11-04-13 03:32 AM, Anisse Taleb wrote:
>> Hi,
>> Please find attached a first draft of a test plan of the IETF codec
>> (Opus).
>> The proposal does not claim to be complete, there are still many
>> missing things, e.g. tandeming cases, tests with delay jitter, dtx
>> etc. Consider it as a starting point for discussion where everyone is
>> welcome to contribute in a constructive manner. Further updates are
>> planned, but let's see first some initial comments.
>>
>> The attachment is a pdf version, please let me know if you would like
>> to see another format and I would be glad to oblige.
>>
>> Comments and additions are welcome!
>>
>> Kind regards,
>> /Anisse
>> (From La Jolla - San Diego).
>>
>>
>>
>> _______________________________________________
>> codec mailing list
>> codec@ietf.org
>> https://www.ietf.org/mailman/listinfo/codec
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec
>
>