Re: [codec] Discussion around ITU LS

Gregory Maxwell <gmaxwell@juniper.net> Wed, 07 September 2011 06:19 UTC

Return-Path: <gmaxwell@juniper.net>
X-Original-To: codec@ietfa.amsl.com
Delivered-To: codec@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8BCA421F8C8F for <codec@ietfa.amsl.com>; Tue, 6 Sep 2011 23:19:01 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.406
X-Spam-Level:
X-Spam-Status: No, score=-5.406 tagged_above=-999 required=5 tests=[AWL=-1.107, BAYES_00=-2.599, MANGLED_SEX=2.3, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PQSwmQ+S8Z3G for <codec@ietfa.amsl.com>; Tue, 6 Sep 2011 23:18:57 -0700 (PDT)
Received: from exprod7og127.obsmtp.com (exprod7og127.obsmtp.com [64.18.2.210]) by ietfa.amsl.com (Postfix) with ESMTP id C3EDF21F845C for <codec@ietf.org>; Tue, 6 Sep 2011 23:18:56 -0700 (PDT)
Received: from P-EMHUB01-HQ.jnpr.net ([66.129.224.36]) (using TLSv1) by exprod7ob127.postini.com ([64.18.6.12]) with SMTP ID DSNKTmcNONlxRoKBe6MTRX5x+KbK2O6jfxPY@postini.com; Tue, 06 Sep 2011 23:20:45 PDT
Received: from EMBX01-HQ.jnpr.net ([fe80::c821:7c81:f21f:8bc7]) by P-EMHUB01-HQ.jnpr.net ([fe80::fc92:eb1:759:2c72%11]) with mapi; Tue, 6 Sep 2011 23:18:21 -0700
From: Gregory Maxwell <gmaxwell@juniper.net>
To: Jean-Marc Valin <jmvalin@mozilla.com>, Cullen Jennings <fluffy@cisco.com>
Date: Tue, 6 Sep 2011 23:18:20 -0700
Thread-Topic: [codec] Discussion around ITU LS
Thread-Index: AcxtFXg+O7jSBM5RRvKAqNTtZRTt1wAAGfU3
Message-ID: <BCB3F026FAC4C145A4A3330806FEFDA93CEC46CD6F@EMBX01-HQ.jnpr.net>
References: <35921B63-3FBC-411D-B587-4AB81F218E57@cisco.com>, <4E66F111.9070008@mozilla.com>
In-Reply-To: <4E66F111.9070008@mozilla.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: Jonathan Rosenberg <jonathan.rosenberg@skype.net>, "codec@ietf.org" <codec@ietf.org>
Subject: Re: [codec] Discussion around ITU LS
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 07 Sep 2011 06:19:01 -0000

IJean-Marc Valin [jmvalin@mozilla.com]:
> believe the comment about portability was mostly referring to Windows
> project files. for the sake of simplicity, those are not included in the
> draft, though there is a separate package that includes them.

One unfortunate thing about writing highly portable code is that
you end up with a bunch of per-platform configuration which complicates
building the software. The draft includes a basic makefile which should
work on a large cross section of systems, but for everything else some
assembly (in the form of writing a project/makefile) is required.

The autoconf and MSVC projects do make things easier on a broader
set of systems, but they're about half the size of the whole codec so not really
suitable for inclusion in the draft.

But they're in the SCM linked from http://opus-codec.org/

The web interface there will build tarfile snapshots from the repository, 
e.g. http://git.xiph.org/?p=opus.git;a=snapshot;h=HEAD;sf=tgz

> As a note
> on portability, Opus has been successfully compiled and run on quite a
> few architectures. I think Greg is in a better position to comment on this.

I do regular testing on the following platforms:
* Linux (x86, x86_64, IA64, PPC, Armv7) (GCC 4.7, 4.5, 3.x depending on the platform)
* Linux with LLVM compiler (x86, x86_64)
* NetBSD (x86)
* FreeBSD (x86)
* Solaris 10 (Ultrasparc)
* Win32 via Mingw
* Win32 via LCC-Win32
* Win32 via OpenWatcom
* Dos32 via OpenWatcom
* IBM S/390
* VAX (MicroVAX 3900, via SIMH, really :) It's quite slow)
(Plus a couple other oddball compilers that I don't test regularly)

None of these require any source modification. I test fixed and floating point builds.
 
I'm also aware of other people who regularly build on OSX (with both llvm-gcc-4.2, and clang 2.1) and with MSVC 

>> What would people like to see with respect to test vectors?
> Again, Greg's been working on that.

For conformance testing my main focus has be on constructing short vectors with complete line-of-code coverage in the decode (save some API and system error handling that can't be reached from the bitstream and non-normative features like the PLC).  I'm pretty close to that now, except for a couple of corner cases in the LPC mode.  I'm planning on testing against bugs we had prior to the bitstream freeze and making sure the comparison tool successfully rejects in those cases.

I'm certainly interested in what else other people are interested in seeing.

I've also been working on "diagnostic" test vectors, which are setup to hit likely failure points in applications (e.g. frame size switches) in a way that will produce audible glitches if the implementation isn't just right.  My thinking is that some people may wish to test embedded implementations which can't be easily put in a full test harness but which you could send a stream to and listen to the output.  Right now the primary impediment on this front is a remaining encoder bug in the reference implementation which cause glitches for my very burdensome test signal at mono/stereo and 10/20ms boundaries. I'm expecting a fix from Koen sometime soon.

Specific to the reference implementation, and separate from the test vectors, I'm also working on a full system test program which can be distributed as part of the source that tests all the API and will give us good coverage of the whole system as part of the build process without having to pack 50 megabytes of vectors in the source directly.   I'll probably be committing this to the SCM later this week.