Re: [codec] draft test and processing plan for the IETF Codec

Jean-Marc Valin <> Fri, 15 April 2011 04:50 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id C0BB7E06AE for <>; Thu, 14 Apr 2011 21:50:15 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id ULmEEtYIpp7T for <>; Thu, 14 Apr 2011 21:50:13 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id 56BD8E0668 for <>; Thu, 14 Apr 2011 21:50:13 -0700 (PDT)
MIME-version: 1.0
Content-transfer-encoding: 7bit
Content-type: text/plain; charset="ISO-8859-1"; format="flowed"
Received: from [] ([]) by (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec 16 2008; 32bit)) with ESMTP id <> for; Fri, 15 Apr 2011 00:48:36 -0400 (EDT)
Message-id: <>
Date: Fri, 15 Apr 2011 00:49:58 -0400
From: Jean-Marc Valin <>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20110223 Thunderbird/3.1.8
To: Jean-Marc Valin <>
References: <> <>
In-reply-to: <>
Cc: "" <>
Subject: Re: [codec] draft test and processing plan for the IETF Codec
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 15 Apr 2011 04:50:15 -0000

So here's some more specific comments on actual bitrates:

1) For narrowband Speex, the rates currently listed are 8, 12, 16 kb/s. 
Those should be changed to 8, 11, 15 kb/s to match the actual Speex 

2) For iLBC, the rates currently listed are 8, 12, 16 kb/s. I think we 
should only use 15.2 kb/s for iLBC. There's another rate, which is 13.33 
kb/s but that's for 30 ms frames so it's not very interesting.

3) For Speex wideband, the rates currently listed are 12, 24, 32 kb/s. I 
think Speex wideband around 12 kb/s is just crap. Worth testing would be 
20.6 and 27.8 kb/s.

4) For super-wideband Speex, I recommend just dumping that. This Speex 
mode was a mistake right from the start and usually has worse quality 
than wideband Speex.

Regarding super-wideband, one thing to keep in mind is that Opus defines 
super-wideband as having a 12 kHz audio bandwidth (24 kHz sampling 
rate). This makes comparisons with other codecs more difficult. The 
rates currently listed for super-wideband are 24, 32, 64 kb/s. I 
recommend running 24 kb/s in super-wideband and running 32 and 64 kb/s 
in fullband mode (even if the input is a 32 kHz signal).

For the very low delay tests (10 ms frame size), I think all the listed 
rates should be using fullband mode except the 32 kb/s.

That's it for now. Any thoughts?


On 11-04-14 11:16 PM, Jean-Marc Valin wrote:
> Hi Anisse,
> I gave some more thought on your proposed test plan and as Cullen
> suggested, I think the main cause of disagreement is not that much on
> the testing, but on the conditions for publishing (large number of BT,
> NWT). Considering that ultimately, the decision to publish a spec is
> always based on WG consensus, then I think that problem can be
> completely bypassed. Once we make it up to the individuals to decide,
> then we can focus on "simply" designing a good test.
> Overall I thought the conditions you were proposing in section 2 were
> pretty reasonable. There's a few details like selecting existing rates
> for codecs like Speex and iLBC, but that should be easy to solve. Once
> these are sorted out, interested parties (we had several hands raised in
> the last meeting) can start testing and we then let each individual
> decide on whether the codec is any good based on the results of the tests.
> Sounds like a plan?
> Jean-Marc
> On 11-04-13 03:32 AM, Anisse Taleb wrote:
>> Hi,
>> Please find attached a first draft of a test plan of the IETF codec
>> (Opus).
>> The proposal does not claim to be complete, there are still many
>> missing things, e.g. tandeming cases, tests with delay jitter, dtx
>> etc. Consider it as a starting point for discussion where everyone is
>> welcome to contribute in a constructive manner. Further updates are
>> planned, but let's see first some initial comments.
>> The attachment is a pdf version, please let me know if you would like
>> to see another format and I would be glad to oblige.
>> Comments and additions are welcome!
>> Kind regards,
>> /Anisse
>> (From La Jolla - San Diego).
>> _______________________________________________
>> codec mailing list
> _______________________________________________
> codec mailing list