Re: [codec] draft test and processing plan for the IETF Codec

[Cullen Jennings <[e-mail]>]

One other little detail where I should have been, uh less fluffy, in my email is the following... It's not really the working group process here, it's just the normal IETF process. There is nothing special about the process for this WG. 

One other random idea ... I wonder if it might be worst lists the tests as priority 1,2, and 3. With the idea that priority 1 tests are ones where people are most interested in the results and 3 the least to help decide what order we run tests in. Its seems to me some tests are far more interesting than others. A test might be uninteresting because we pretty much already know what the result will be or it might be less interesting just because it's a bit scenario that will not be used as much. Just a random idea I am tossing out there, no idea if this would help or not. 

Cullen


On Apr 15, 2011, at 4:41 AM, Jean-Marc Valin wrote:

> Koen,
> 
> The point I was making in the earlier email and that Cullen has stated earlier is that we don't even need *any* BT or NWT requirements in this document. It will be up to the WG individual participants to decide for themselves whether they think it's good to publish Opus, based on the information available at the time they make the decision. So Anisse's test plan proposal is meant to "gather useful data", not decide on whether to publish.
> 
> 	Jean-Marc
> 
> On 11-04-15 04:44 AM, Koen Vos wrote:
>> I would also suggest replacing all BT (better than) requirements by NWT (no worse than).
>> 
>> My reasoning is that:
>> - The WG never had the goal to be better than other codecs (see charter).
>> - Proving to be better can be very hard, especially when several codecs are close to transparent.  To show significance in that case you'd need a vast number of listeners, which makes a test more cumbersome to perform.
>> 
>> best,
>> koen.
>> 
>> 
>> ----- Original Message -----
>> From: "Jean-Marc Valin"<[e-mail]>
>> To: "Jean-Marc Valin"<[e-mail]>
>> Cc: [e-mail]
>> Sent: Thursday, April 14, 2011 9:49:58 PM
>> Subject: Re: [codec] draft test and processing plan for the IETF Codec
>> 
>> So here's some more specific comments on actual bitrates:
>> 
>> 1) For narrowband Speex, the rates currently listed are 8, 12, 16 kb/s.
>> Those should be changed to 8, 11, 15 kb/s to match the actual Speex
>> bitrates.
>> 
>> 2) For iLBC, the rates currently listed are 8, 12, 16 kb/s. I think we
>> should only use 15.2 kb/s for iLBC. There's another rate, which is 13.33
>> kb/s but that's for 30 ms frames so it's not very interesting.
>> 
>> 3) For Speex wideband, the rates currently listed are 12, 24, 32 kb/s. I
>> think Speex wideband around 12 kb/s is just crap. Worth testing would be
>> 20.6 and 27.8 kb/s.
>> 
>> 4) For super-wideband Speex, I recommend just dumping that. This Speex
>> mode was a mistake right from the start and usually has worse quality
>> than wideband Speex.
>> 
>> Regarding super-wideband, one thing to keep in mind is that Opus defines
>> super-wideband as having a 12 kHz audio bandwidth (24 kHz sampling
>> rate). This makes comparisons with other codecs more difficult. The
>> rates currently listed for super-wideband are 24, 32, 64 kb/s. I
>> recommend running 24 kb/s in super-wideband and running 32 and 64 kb/s
>> in fullband mode (even if the input is a 32 kHz signal).
>> 
>> For the very low delay tests (10 ms frame size), I think all the listed
>> rates should be using fullband mode except the 32 kb/s.
>> 
>> That's it for now. Any thoughts?
>> 
>> 	Jean-Marc
>> 
>> 
>> On 11-04-14 11:16 PM, Jean-Marc Valin wrote:
>>> Hi Anisse,
>>> 
>>> I gave some more thought on your proposed test plan and as Cullen
>>> suggested, I think the main cause of disagreement is not that much on
>>> the testing, but on the conditions for publishing (large number of BT,
>>> NWT). Considering that ultimately, the decision to publish a spec is
>>> always based on WG consensus, then I think that problem can be
>>> completely bypassed. Once we make it up to the individuals to decide,
>>> then we can focus on "simply" designing a good test.
>>> 
>>> Overall I thought the conditions you were proposing in section 2 were
>>> pretty reasonable. There's a few details like selecting existing rates
>>> for codecs like Speex and iLBC, but that should be easy to solve. Once
>>> these are sorted out, interested parties (we had several hands raised in
>>> the last meeting) can start testing and we then let each individual
>>> decide on whether the codec is any good based on the results of the tests.
>>> 
>>> Sounds like a plan?
>>> 
>>> Jean-Marc
>>> 
>>> 
>>> On 11-04-13 03:32 AM, Anisse Taleb wrote:
>>>> Hi,
>>>> Please find attached a first draft of a test plan of the IETF codec
>>>> (Opus).
>>>> The proposal does not claim to be complete, there are still many
>>>> missing things, e.g. tandeming cases, tests with delay jitter, dtx
>>>> etc. Consider it as a starting point for discussion where everyone is
>>>> welcome to contribute in a constructive manner. Further updates are
>>>> planned, but let's see first some initial comments.
>>>> 
>>>> The attachment is a pdf version, please let me know if you would like
>>>> to see another format and I would be glad to oblige.
>>>> 
>>>> Comments and additions are welcome!
>>>> 
>>>> Kind regards,
>>>> /Anisse
>>>> (From La Jolla - San Diego).
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> codec mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/codec
>>> _______________________________________________
>>> codec mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/codec
>>> 
>>> 
>> _______________________________________________
>> codec mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/codec
>> 
>> 
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec