Re: [codec] A concrete proposal for requirements and testing

Roman Shpount <roman@telurix.com> Thu, 31 March 2011 16:18 UTC

Return-Path: <roman@telurix.com>
X-Original-To: codec@core3.amsl.com
Delivered-To: codec@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 3444E3A69BF for <codec@core3.amsl.com>; Thu, 31 Mar 2011 09:18:03 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.976
X-Spam-Level:
X-Spam-Status: No, score=-2.976 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jjFoDbVXHrlc for <codec@core3.amsl.com>; Thu, 31 Mar 2011 09:18:02 -0700 (PDT)
Received: from mail-iw0-f172.google.com (mail-iw0-f172.google.com [209.85.214.172]) by core3.amsl.com (Postfix) with ESMTP id EDA553A684E for <codec@ietf.org>; Thu, 31 Mar 2011 09:18:01 -0700 (PDT)
Received: by iwn39 with SMTP id 39so2949449iwn.31 for <codec@ietf.org>; Thu, 31 Mar 2011 09:19:41 -0700 (PDT)
Received: by 10.42.221.136 with SMTP id ic8mr1318062icb.487.1301588381510; Thu, 31 Mar 2011 09:19:41 -0700 (PDT)
Received: from mail-iy0-f172.google.com (mail-iy0-f172.google.com [209.85.210.172]) by mx.google.com with ESMTPS id i20sm831785iby.65.2011.03.31.09.19.39 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 31 Mar 2011 09:19:40 -0700 (PDT)
Received: by iye19 with SMTP id 19so2955090iye.31 for <codec@ietf.org>; Thu, 31 Mar 2011 09:19:39 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.194.34 with SMTP id dw34mr2573415ibb.117.1301588379206; Thu, 31 Mar 2011 09:19:39 -0700 (PDT)
Received: by 10.231.19.11 with HTTP; Thu, 31 Mar 2011 09:19:39 -0700 (PDT)
In-Reply-To: <64212FE1AE068044AD567CCB214073F123A10234@MAIL2.octasic.com>
References: <64212FE1AE068044AD567CCB214073F123A10234@MAIL2.octasic.com>
Date: Thu, 31 Mar 2011 12:19:39 -0400
Message-ID: <AANLkTinrS6e+i_VE5Q91gbNV5yhwi3Ad+wZOUsKYGNZn@mail.gmail.com>
From: Roman Shpount <roman@telurix.com>
To: Jean-Marc Valin <jean-marc.valin@octasic.com>
Content-Type: multipart/alternative; boundary="005045016245fc828a049fc9a8bc"
Cc: "codec@ietf.org" <codec@ietf.org>
Subject: Re: [codec] A concrete proposal for requirements and testing
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 31 Mar 2011 16:18:03 -0000

I might be missing something, but how is it that G.722 is clearly
out-performed by G.722.1? As far as I know G.722 at 64 KB has higher quality
then wideband G.722.1 at any rate. Obviously, G.722 has higher bandwidth
requirements, but has higher quality and lower bandwidth. Outperforming
G.722 at 16KHz for all languages, and especially for multiple simultaneous
speakers or non-speech sources is not an easy thing and should be tested
against.
_____________
Roman Shpount


On Thu, Mar 31, 2011 at 10:53 AM, Jean-Marc Valin <
jean-marc.valin@octasic.com> wrote:

>  Hi,
>
> Following the meeting and post-meeting discussions about requirements and
> testing, we would like to make the following proposal which addresses the
> opposing views which prevented consensus in the meeting today.
>
> First, we propose to remove the following codecs from the requirements:
>
> - GSM-FR, based on consensus from the list
> - G.722, based on being clearly out-performed by G.722.1
> - Speex-UWB, based on the fact that the author himself does not recommend
> it being used :-)
>
> We can keep the other reference codecs as minimum quality requirement and
> include being no worse than AMR-NB and AMR-WB as "objectives" that are "nice
> to have", but not hard requirements.
>
> From there and based on the listening tests presented by Jan Skoglund
> today, let's see what we can already conclude and what still needs more
> testing:
>
> 1) The narrowband test showed that Opus had higher quality than Speex at 11
> kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2
> requirement of out-performing Speex in narrowband mode?
>
> 2) The narrowband test showed that Opus had higher quality at 11 kb/s than
> iLBC at 15 kb/s. Does anyone disagree that this is sufficient to meet the
> Sec 4.2 requirement of out-performing iLBC.
>
> 3) There have been no formal comparison with AMR-NB yet. What do you think
> would be sufficient to assess the quality of Opus compared to AMR-NB?
>
> 4) The wideband test showed that Opus at 19.85 kb/s had higher quality than
> Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient to meet
> the Sec 4.2 requirement of out-performing Speex in wideband mode?
>
> 5) The wideband test showed that Opus at 19.85 kb/s had higher quality than
> G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to meet the
> Sec 4.2 requirement of out-performing G.722.1?
>
> 6) The wideband test showed that Opus at 19.85 kb/s had higher quality than
> AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient to
> concluded that the proposed "nice to have" objective of "no worse than
> AMR-WB" is met?
>
> 7) The fullband test showed that Opus at 32 kb/s had higher quality than
> G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet the
> Sec 4.2 requirement of out-performing G.722.1C, considering that G.719 has
> already been shown to out-perform G.722.1C
>
> If you disagree with any of the points above -- as may very well be the
> case -- please do provide a concrete test proposal that would be sufficient
> to convince you.
>
> Cheers,
>
>     Jean-Marc and Koen
>
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec
>
>