Re: [codec] draft test and processing plan for the IETF Codec

Stephen Botzko <> Wed, 13 April 2011 14:13 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 70594E0699 for <>; Wed, 13 Apr 2011 07:13:53 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -3.148
X-Spam-Status: No, score=-3.148 tagged_above=-999 required=5 tests=[AWL=0.450, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id bB89gr8bmlDd for <>; Wed, 13 Apr 2011 07:13:52 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id 78C1FE0690 for <>; Wed, 13 Apr 2011 07:13:52 -0700 (PDT)
Received: by vws12 with SMTP id 12so609087vws.31 for <>; Wed, 13 Apr 2011 07:13:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=6zpWHK/5fvpeXRY2qF+O1ZZvlbr4y+mMOvScjByUT4c=; b=jk/CMJnPTmDFkShMlASadtdei7VTDsMm05quwIzq6UWSzfXQW32MyphxmAudxvc1b0 ZaAbOqtwr893vonBEU0ibVimzrciYHsPdaXYFtp2d/dS1jdwg8pa1B8ML2xFKF2fMpnb atD+wXPURq2Rp2E+rgvE06ZjxLIdDTv1Rs554=
DomainKey-Signature: a=rsa-sha1; c=nofws;; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=ElwA973mP8PCsakDPPzUzHivD5Xjj3zfh0Znrq2+kxIwjViE/Qc/cX152hA/nDyHCF b8LWVtvBz8k1OA9ZBGcxpSdYMAF0tmSa70c1VLLrn+MfhRg5wPWlgh0EKazBYDFv7QAV fv0b4qZnOQrEujISQMSv9SSxTtxxBIUip6BKk=
MIME-Version: 1.0
Received: by with SMTP id bp6mr1348999vdc.90.1302704032007; Wed, 13 Apr 2011 07:13:52 -0700 (PDT)
Received: by with HTTP; Wed, 13 Apr 2011 07:13:51 -0700 (PDT)
In-Reply-To: <>
References: <> <>
Date: Wed, 13 Apr 2011 10:13:51 -0400
Message-ID: <>
From: Stephen Botzko <>
Content-Type: multipart/alternative; boundary="bcaec51b195113463a04a0cd6bd3"
Cc: "" <>
Subject: Re: [codec] draft test and processing plan for the IETF Codec
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 13 Apr 2011 14:13:53 -0000

We are not yet at WG LC, and the RFC publication process itself takes many
months.  So I am not seeing the scheduling issues as insurmountable. We
could easily require some core tests prior to WG LC, and continue testing
during the publication process.

I think the best course right now is to refine the test plan, and sort out
the process steps/scheduling when we have stronger consensus on the test
plan itself.

I agree that the tests need to be run on the version we plan to submit for
publication.  Making improvements later on does not have to mean the entire
test suite needs to be re-run - subsequent testing could be targeted to test
the changes.

Stephen Botzko

On Wed, Apr 13, 2011 at 9:38 AM, Benjamin M. Schwartz <> wrote:

> On 04/13/2011 03:32 AM, Anisse Taleb wrote:
> > Please find attached a first draft of a test plan of the IETF codec
> (Opus).
> Thank you for drawing up this test plan, which clearly required a great
> deal of thought.  The results of such testing would certainly be very
> interesting to many.
> However, I think the execution of such a test is clearly _not_ an
> appropriate prerequisite for publishing a Proposed Standard.  By my
> calculations, the draft plan presently calls for over 1300 hours of
> listening tests, counting only audio being played, estimating 10-second
> samples and the minimum number of listeners.  Even if many listeners are
> listening in parallel, and overheads (such as delays between samples) are
> low, conducting such a test would still take many months.
> Such an extensive, expensive battery of tests can hardly be justified on
> some arbitrary codec version still under development.  It can only be
> justified if the codec being tested is not going to change, so that the
> sponsoring organizations can use the results to determine whether the
> codec meets their performance goals.
> Let's standardize, and then invite ultra-comprehensive systematic
> characterization.
> --Ben
> _______________________________________________
> codec mailing list