Re: [codec] Listening tests at Google

[Jan Skoglund <[e-mail]>]

Hey,

Sure,

1) For narrowband Mandarin speech Opus at 11 kbps is comparable to
iLBC at 15 kbps and superior to Speex at 11 kbps. In the previous
tests with English speech Opus outperformed them both, so there was a
(small) difference.
For wideband/fullband speech a striking observation is that Opus at 32
kbps is (again) almost transparent, although there is a small and
statistically significant difference from the fullband reference
material. Opus at 32 kbps is also better than G.719 at the same bit
rate.  Opus at 20 kbps is not only better than Speex at 24 kbps and
G.722.1 at 24 kbps, it is also better than the reference signal
low-passed at 7 kHz. This is likely due to its high quality and its
higher signal bandwidth (8 kHz). These results are pretty consistent
with the previous results for English speech, with the difference that
in the previous test Opus at 20 kbps outperformed G.719 at 32 kbps,
which it did not this time.

2) Pre-coding Opus NB with G.711 is comparable to pre-coding AMR NB
with G.711. Adding one stage of transcoding reduces the performance
slightly (although statistically significant). Transcoding AMR NB with
itself through G.711 is worse than transcoding AMR NB to Opus through
G.711 or the other way around.
Opus WB at 19.85 kbps is again better than speech LP-filtered at 7.0
kHz,  as in the Mandarin and English speech tests. Single-coded AMR WB
is significantly worse than single-coded Opus WB, also consistent with
previous tests. Transcoding Opus and AMR WB is only slightly worse
than AMR WB, although the difference is statistically significant.

Jan


On Tue, Oct 18, 2011 at 7:01 PM, Paul Beaumont
<[e-mail]> wrote:
> Hi Jan
>
> Thank you for performing and sharing the results of your testing.
>
> Please can you summarise what you see as the key findings wrt ...
>
> 1). Opus on Mandarin speech - be good to baseline this against comparable
> English speech results and
> 2). Opus when Transcoded/Tandemed
>
> Regards
> Paul
>
> --
>
> On 14/09/2011 06:59, "Jan Skoglund" <[e-mail]> wrote:
>
>>Hey,
>>
>>The listening tests at Google are now done and here are the results.
>>
>>Regards,
>>Jan
>>
>>
>>On Fri, Aug 19, 2011 at 3:28 PM, Jan Skoglund <[e-mail]> wrote:
>>> Hey,
>>>
>>> We've prepared some listening tests for Opus on Mandarin speech and
>>> transcoding (English) speech at Google. We started the tests today and
>>> will have results at the end of August. The tests are split in four
>>> MUSHRA groups:
>>>
>>> Test 1a - Mandarin NB
>>> --------------------------
>>> Opus NB
>>> Speex NB
>>> iLBC NB
>>> Reference and 2 anchors (LP 3.5 kHz and MNRU of 15 dB)
>>>
>>> Test 1b - Mandarin "HD Voice" (WB, FB)
>>> --------------------------------------------------------
>>> Opus FB
>>> G.719 FB
>>> Opus WB
>>> Speex WB
>>> G.722.1 WB
>>> Reference and 2 anchors (LP 3.5 kHz and LP 7 kHz)
>>>
>>> Test 2a - Transcoding NB
>>> -------------------------------
>>> G.711 -> Opus NB
>>> G.711 -> AMR-NB
>>> Opus NB -> G.711 -> AMR-NB
>>> AMR-NB -> G.711 -> Opus NB
>>> AMR-NB -> G.711 -> AMR-NB
>>> Reference and 2 anchors (LP 3.5 kHz and MNRU of 15 dB)
>>>
>>> Test 2b - Transcoding WB
>>> --------------------------------
>>> Opus WB
>>> AMR-WB
>>> Opus WB -> AMR-WB
>>> AMR-WB -> Opus WB
>>> Reference and 2 anchors (LP 3.5 kHz and LP 7 kHz)
>>>
>>>
>>> Cheers,
>>> Jan
>>>
>>_______________________________________________
>>codec mailing list
>>[e-mail]
>>https://www.ietf.org/mailman/listinfo/codec
>
>
>