Re: [codec] Tübingen Test plan

"Christian Hoene" <hoene@uni-tuebingen.de> Mon, 15 August 2011 08:26 UTC

Return-Path: <hoene@uni-tuebingen.de>
X-Original-To: codec@ietfa.amsl.com
Delivered-To: codec@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1EFE921F8ABD for <codec@ietfa.amsl.com>; Mon, 15 Aug 2011 01:26:17 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.796
X-Spam-Level:
X-Spam-Status: No, score=-2.796 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HELO_EQ_DE=0.35, HTML_MESSAGE=0.001, J_CHICKENPOX_42=0.6, J_CHICKENPOX_52=0.6, J_CHICKENPOX_62=0.6, J_CHICKENPOX_72=0.6, J_CHICKENPOX_82=0.6, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_MED=-4, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pOE3Oj4kAo6k for <codec@ietfa.amsl.com>; Mon, 15 Aug 2011 01:26:15 -0700 (PDT)
Received: from mx05.uni-tuebingen.de (mx05.uni-tuebingen.de [134.2.3.4]) by ietfa.amsl.com (Postfix) with ESMTP id 91F2021F8A6C for <codec@ietf.org>; Mon, 15 Aug 2011 01:26:14 -0700 (PDT)
Received: from hoeneT60 (stgt-d9be5f78.pool.mediaWays.net [217.190.95.120]) (authenticated bits=0) by mx05.uni-tuebingen.de (8.13.6/8.13.6) with ESMTP id p7F8QhOJ020943 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Mon, 15 Aug 2011 10:26:50 +0200
From: "Christian Hoene" <hoene@uni-tuebingen.de>
To: "'Koen Vos'" <koen.vos@skype.net>
References: <000d01cc5ab9$30ef71f0$92ce55d0$@uni-tuebingen.de> <1586180885.47515.1313353689004.JavaMail.root@lu2-zimbra>
In-Reply-To: <1586180885.47515.1313353689004.JavaMail.root@lu2-zimbra>
Date: Mon, 15 Aug 2011 10:26:48 +0200
Organization: =?utf-8?Q?Universit=C3=A4t_T=C3=BCbingen?=
Message-ID: <005501cc5b25$17e03930$47a0ab90$@uni-tuebingen.de>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="----=_NextPart_000_0056_01CC5B35.DB7B31A0"
X-Mailer: Microsoft Outlook 14.0
Thread-Index: AQLqueMa9oU8AxHndCexpmnef+6joZLg1eTw
Content-Language: de
X-AntiVirus: NOT checked by Avira MailGate (version: 3.2.1.23; host: mx05)
Cc: codec@ietf.org
Subject: Re: [codec] =?utf-8?q?T=C3=BCbingen_Test_plan?=
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 15 Aug 2011 08:26:17 -0000

Hi Koen,

 

What audio bandwidths are you planning to use for the Opus test cases?

[Christian Hoene] Actually, Opus selects the mode by itself. I have printed out the following data

Mode CELT bitRate 64000 fs 48000 channels 2 framesize 960

Mode CELT bitRate 96000 fs 48000 channels 2 framesize 120

Mode HYBRID bitRate 32000 fs 48000 channels 2 framesize 960

Mode SILK bitRate 16000 fs 48000 channels 2 framesize 960

With best regards,

 

Christian Hoene

 

 

  _____  

From: "Christian Hoene" <hoene@uni-tuebingen.de>
To: codec@ietf.org
Sent: Sunday, August 14, 2011 12:34:31 PM
Subject: Re: [codec] Tübingen Test plan




Hi,

 

Thanks for the input. I have updated the test plan:

 

 

Testplan for MUSHRA Listening Tests 
Focusing on a Conferencing Scenario

Mansoor Hyder, Christian Hoene
Universität Tübingen
14. August 2011
Version 2


Test setup


MUSHRA tests following the recommendation ITU-R BS 1584.1 will take place on August 16th, from 9:00 till 19:00 in the office B303/B304 at the building Sand 13 in Tübingen, Germany.

Tests will be done with Sennheiser HD 280 pro headphones, a PC and the software „RateIt“ version 0.1.

Twenty participants have been selected from employees and students of the University of Tübingen. They have well to very good hearing abilities. Participant will receive soft drinks and/or coffee and 8 € compensation. 

Each session will last at most 60 minutes. This includes 15 to 20 minutes for explaining and practicing the tests, 25 to 30 minutes of testing, and 5 to 10 minutes of questioning and farewell.

In total, 40 stimuli will be tested (in addition to the references).


Test stimuli


The focus of the stimuli is on a conferencing scenario, in which multiple persons speak at the same time. Only stereo test stimuli will be taken and only stereo coding modes will be tested.

Five different reference samples will be taken:

1.       Female German voice (stereo)

2.       Two female voices (stereo, cross talking)

3.       One female voice (binaural rendered, moving)

4.       Two female voices (binaural rendered, cross talking)

5.       Music (an acappella  song by “Die Prinzen”)

The following codec and codec modes will be tests

1.       Opus, Silk mode, stereo, high complexity, 20ms frame size, at 16 kbps.

2.       Opus, Hybrid mode, stereo, high complexity, 20ms frame size, at 32 kbps

3.       Opus, Celt mode, high complexity, stereo, 20ms frame size at 64 kbps

4.       Opus, Celt mode, high complexity, stereo, 5ms frame size, at 96 kbps

5.       G.719, stereo, at 2x32 kbps (Fix-point-200806-Release-1.0a)

6.       AMR-WB+, stereo, at 16 kbps (Code from TS 26304 Version 6.6)

7.       AMR-WB+, stereo, at 32 kbps (Code from TS 26304 Version 6.6)

8.        TBD, stereo at 64 kbps, 20ms frame size

In total, 5x8=40 stimuli are going to be tested.


Script to make stimuli


 

#!/bin/bash

OPUS=../opus.testing/draft-ietf-codec-opus-07/test_opus

G719e=../G.719-Fix-point-200806-Release-1.0a/out/encoder 

G719d=../G.719-Fix-point-200806-Release-1.0a/out/decoder

AMRWBPe="wine ../26304_ANSI-C_source_code_v6_6_0/encoder"

AMRWBPd="wine ../26304_ANSI-C_source_code_v6_6_0/decoder"

 

I=1

for file in ../reference\ items/*\.wav

do

    echo $I $file

 

    sox "$file" -r48000 -c2 input.sw

    $OPUS 0 48000 2 16000 -cbr -framesize 20 -bandwidth WB input.sw output.sw >/dev/null

    sox -r48000 -c2 output.sw $I-opus.16k.wav 

    $OPUS 0 48000 2 32000 -cbr -framesize 20 -bandwidth FB input.sw output.sw >/dev/null

    sox -r48000 -c2 output.sw $I-opus.32k.wav

    $OPUS 1 48000 2 64000 -cbr -framesize 20 -bandwidth FB input.sw output.sw >/dev/null

    sox -r48000 -c2 output.sw $I-opus.64k.wav

    $OPUS 1 48000 2 96000 -cbr -framesize 2.5 -bandwidth FB input.sw output.sw >/dev/null

    sox -r48000 -c2 output.sw $I-opus.96k.wav

    rm output.sw

 

    sox -r48000 -c2 input.sw -r48000 -c1 left.sw remix 1 0

    sox -r48000 -c2 input.sw -r48000 -c1 right.sw remix 0 1

    $G719e -r 32000 -i left.sw -o left.g192

    $G719e -r 32000 -i right.sw -o right.g192

    $G719d -i left.g192 -o outleft.sw

    $G719d -i right.g192 -o outright.sw

    sox -r48000 -c1 outleft.sw -c2 outleft.wav remix 1 0 

    sox -r48000 -c1 outright.sw -c2 outright.wav remix 0 1

    sox -m outleft.wav outright.wav $I-g719.64k.wav

    rm left.sw right.sw left.g192 right.g192 outleft.sw outright.sw 

    rm outleft.wav outright.wav input.sw

 

    $AMRWBPe -rate 16 -if "$file" -of out.wb+

    $AMRWBPd  -if out.wb+ -of $I-amrwbp.16k.wav

 

    $AMRWBPe -rate 32 -if "$file" -of out.wb+

    $AMRWBPd -if out.wb+ -of $I-amrwbp.32k.wav

    rm out.wb+

 

    sox "$file" -r48000 -c2 $I-reference.wav

    sox "$file" -r48000 -c2 $I-hidden.wav

    sox "$file" -r48000 -c2 $I-anchor.wav lowpass 3500

 

    let I=$I+1

done

 

 

 

---------------------------------------------------------------------------------

Dr.-Ing. Christian Hoene, University of Tübingen, Computer Science, Chair of

Communication Networks, Research Group Interactive Communication Systems (ICS)

Sand 13, 72076 Tübingen, Germany, Tel +49 7071 2970532, www.net.uni-tuebingen.de <http://www.net.uni-tuebingen.de/> 

 

From: codec-bounces@ietf.org [mailto:codec-bounces@ietf.org] On Behalf Of Christian Hoene
Sent: Sunday, August 14, 2011 3:22 PM
To: codec@ietf.org
Subject: [codec] Tübingen Test plan

 

Hi, 

 

I was compiling the following tests plan for Tuesday.

Any comments and hint to improve the test plan?

 

With best regards,

 

Christian

 

 


_______________________________________________
codec mailing list
codec@ietf.org
https://www.ietf.org/mailman/listinfo/codec