Re: [codec] Listening tests at Google

Jan Skoglund <jks@google.com> Thu, 20 October 2011 14:33 UTC

Return-Path: <jks@google.com>
X-Original-To: codec@ietfa.amsl.com
Delivered-To: codec@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8DB6421F8C2F for <codec@ietfa.amsl.com>; Thu, 20 Oct 2011 07:33:11 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -105.977
X-Spam-Level:
X-Spam-Status: No, score=-105.977 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eLVUJ3zjOoKw for <codec@ietfa.amsl.com>; Thu, 20 Oct 2011 07:33:11 -0700 (PDT)
Received: from smtp-out.google.com (smtp-out.google.com [74.125.121.67]) by ietfa.amsl.com (Postfix) with ESMTP id 9DB6E21F8C2B for <codec@ietf.org>; Thu, 20 Oct 2011 07:33:10 -0700 (PDT)
Received: from hpaq5.eem.corp.google.com (hpaq5.eem.corp.google.com [172.25.149.5]) by smtp-out.google.com with ESMTP id p9KEX9An029300 for <codec@ietf.org>; Thu, 20 Oct 2011 07:33:09 -0700
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=google.com; s=beta; t=1319121189; bh=LZ6GPMAz5eblnQ3qHrgEiijeWN4=; h=MIME-Version:In-Reply-To:References:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=f3Jw2F4FZFhIH6EMMLkncifanRews25rEBVONBrnJ2Lshslyft8/W2M8eGQt1ln2g JOu6y2hzVklIEEbF2YljA==
DomainKey-Signature: a=rsa-sha1; s=beta; d=google.com; c=nofws; q=dns; h=dkim-signature:mime-version:in-reply-to:references:from:date: message-id:subject:to:cc:content-type:x-system-of-record; b=snssXSJEIkACTPZZ5V0UL7perVj7b49Zaf9V84Kq5LPiZJtzdqO3jNkyd9MsL25fX UsaH8joO1Q5fGz7LEUdag==
Received: from gyd8 (gyd8.prod.google.com [10.243.49.200]) by hpaq5.eem.corp.google.com with ESMTP id p9KEVBLl017087 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NOT) for <codec@ietf.org>; Thu, 20 Oct 2011 07:33:08 -0700
Received: by gyd8 with SMTP id 8so4042355gyd.2 for <codec@ietf.org>; Thu, 20 Oct 2011 07:33:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=beta; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:x-system-of-record; bh=1mZB556aKn9+aul2odqXjC31rBEojEYVtv7eEFAhdf8=; b=nTs4QyKYak5mdGVKppsdv9otmxPgwceXEXyJgKqhF9n3hGTm6RlZeox9QUTf31rKpc ezEbO56SlxQmm914YkaQ==
Received: by 10.229.86.145 with SMTP id s17mr2469644qcl.36.1319121188572; Thu, 20 Oct 2011 07:33:08 -0700 (PDT)
Received: by 10.229.86.145 with SMTP id s17mr2469633qcl.36.1319121188282; Thu, 20 Oct 2011 07:33:08 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.91.129 with HTTP; Thu, 20 Oct 2011 07:32:48 -0700 (PDT)
In-Reply-To: <CAC3C194.1C4DF%paulbeaumont.ietf@gmail.com>
References: <CA+KMCSWzNUVWNZsvTKsv_+N=A1keUyOhM5qB3yj+GcUEp2VcFQ@mail.gmail.com> <CAC3C194.1C4DF%paulbeaumont.ietf@gmail.com>
From: Jan Skoglund <jks@google.com>
Date: Thu, 20 Oct 2011 10:32:48 -0400
Message-ID: <CA+KMCSW_oasjqqOW+Uta=PHaVyMEKkCd1aTB4Zc8=s2O1k=j1g@mail.gmail.com>
To: Paul Beaumont <paulbeaumont.ietf@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
X-System-Of-Record: true
Cc: IETF - CoDec <codec@ietf.org>
Subject: Re: [codec] Listening tests at Google
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 20 Oct 2011 14:33:11 -0000

Hey,

Sure,

1) For narrowband Mandarin speech Opus at 11 kbps is comparable to
iLBC at 15 kbps and superior to Speex at 11 kbps. In the previous
tests with English speech Opus outperformed them both, so there was a
(small) difference.
For wideband/fullband speech a striking observation is that Opus at 32
kbps is (again) almost transparent, although there is a small and
statistically significant difference from the fullband reference
material. Opus at 32 kbps is also better than G.719 at the same bit
rate.  Opus at 20 kbps is not only better than Speex at 24 kbps and
G.722.1 at 24 kbps, it is also better than the reference signal
low-passed at 7 kHz. This is likely due to its high quality and its
higher signal bandwidth (8 kHz). These results are pretty consistent
with the previous results for English speech, with the difference that
in the previous test Opus at 20 kbps outperformed G.719 at 32 kbps,
which it did not this time.

2) Pre-coding Opus NB with G.711 is comparable to pre-coding AMR NB
with G.711. Adding one stage of transcoding reduces the performance
slightly (although statistically significant). Transcoding AMR NB with
itself through G.711 is worse than transcoding AMR NB to Opus through
G.711 or the other way around.
Opus WB at 19.85 kbps is again better than speech LP-filtered at 7.0
kHz,  as in the Mandarin and English speech tests. Single-coded AMR WB
is significantly worse than single-coded Opus WB, also consistent with
previous tests. Transcoding Opus and AMR WB is only slightly worse
than AMR WB, although the difference is statistically significant.

Jan


On Tue, Oct 18, 2011 at 7:01 PM, Paul Beaumont
<paulbeaumont.ietf@gmail.com> wrote:
> Hi Jan
>
> Thank you for performing and sharing the results of your testing.
>
> Please can you summarise what you see as the key findings wrt ...
>
> 1). Opus on Mandarin speech - be good to baseline this against comparable
> English speech results and
> 2). Opus when Transcoded/Tandemed
>
> Regards
> Paul
>
> --
>
> On 14/09/2011 06:59, "Jan Skoglund" <jks@google.com> wrote:
>
>>Hey,
>>
>>The listening tests at Google are now done and here are the results.
>>
>>Regards,
>>Jan
>>
>>
>>On Fri, Aug 19, 2011 at 3:28 PM, Jan Skoglund <jks@google.com> wrote:
>>> Hey,
>>>
>>> We've prepared some listening tests for Opus on Mandarin speech and
>>> transcoding (English) speech at Google. We started the tests today and
>>> will have results at the end of August. The tests are split in four
>>> MUSHRA groups:
>>>
>>> Test 1a - Mandarin NB
>>> --------------------------
>>> Opus NB
>>> Speex NB
>>> iLBC NB
>>> Reference and 2 anchors (LP 3.5 kHz and MNRU of 15 dB)
>>>
>>> Test 1b - Mandarin "HD Voice" (WB, FB)
>>> --------------------------------------------------------
>>> Opus FB
>>> G.719 FB
>>> Opus WB
>>> Speex WB
>>> G.722.1 WB
>>> Reference and 2 anchors (LP 3.5 kHz and LP 7 kHz)
>>>
>>> Test 2a - Transcoding NB
>>> -------------------------------
>>> G.711 -> Opus NB
>>> G.711 -> AMR-NB
>>> Opus NB -> G.711 -> AMR-NB
>>> AMR-NB -> G.711 -> Opus NB
>>> AMR-NB -> G.711 -> AMR-NB
>>> Reference and 2 anchors (LP 3.5 kHz and MNRU of 15 dB)
>>>
>>> Test 2b - Transcoding WB
>>> --------------------------------
>>> Opus WB
>>> AMR-WB
>>> Opus WB -> AMR-WB
>>> AMR-WB -> Opus WB
>>> Reference and 2 anchors (LP 3.5 kHz and LP 7 kHz)
>>>
>>>
>>> Cheers,
>>> Jan
>>>
>>_______________________________________________
>>codec mailing list
>>codec@ietf.org
>>https://www.ietf.org/mailman/listinfo/codec
>
>
>