Re: [codec] Tübingen Test plan

Koen Vos <koen.vos@skype.net> Sun, 14 August 2011 20:27 UTC

Return-Path: <koen.vos@skype.net>
X-Original-To: codec@ietfa.amsl.com
Delivered-To: codec@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0243D21F8581 for <codec@ietfa.amsl.com>; Sun, 14 Aug 2011 13:27:30 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.854
X-Spam-Level:
X-Spam-Status: No, score=0.854 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, J_CHICKENPOX_42=0.6, J_CHICKENPOX_52=0.6, J_CHICKENPOX_62=0.6, J_CHICKENPOX_72=0.6, J_CHICKENPOX_82=0.6, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IQo398IG1Rpc for <codec@ietfa.amsl.com>; Sun, 14 Aug 2011 13:27:28 -0700 (PDT)
Received: from mx.skype.net (mx.skype.net [78.141.177.88]) by ietfa.amsl.com (Postfix) with ESMTP id 2DB6921F853E for <codec@ietf.org>; Sun, 14 Aug 2011 13:27:28 -0700 (PDT)
Received: from mx.skype.net (localhost [127.0.0.1]) by mx.skype.net (Postfix) with ESMTP id B3F2816E2; Sun, 14 Aug 2011 22:28:10 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=skype.net; h=date:from:to :cc:message-id:in-reply-to:subject:mime-version:content-type; s= mx; bh=ZKb4D4DpqscC5AbgmVr50tblJqM=; b=nRbTevUGFLvwt/O4ojUj2m9Ea hNSN6Cedz25OyAsNjWaP5Pky6QAWbeP26vxiO0pJ9/gbby6qjxGaIga5f7z9W+Vr zYt78GpYRlCmVbVUFlPSZ0z62dWWloCjMlYPu8QZKKrr8gvrya8CAutdSMlwI1Rd 1Mamq98rIO2VHok99c=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=skype.net; h=date:from:to:cc :message-id:in-reply-to:subject:mime-version:content-type; q=dns ; s=mx; b=oSSHr99m3qtOv98U/d1UwbpwrDMfnu0u39PxSklLnltTigLrWF2Ud3 /owd2k8RVPYC+PRZEgzPAH3ycjFmP8VUp3rqpOQndCbfjosBFF98eeR7mDOOBE+W 8vVDZnKxPvgSNaD3x5dXbMx7yUC1hSzlcfk4yNNrBzy0tnzvPw9FY=
Received: from zimbra.skype.net (zimbra.skype.net [78.141.177.82]) by mx.skype.net (Postfix) with ESMTP id B22B67F8; Sun, 14 Aug 2011 22:28:10 +0200 (CEST)
Received: from localhost (localhost [127.0.0.1]) by zimbra.skype.net (Postfix) with ESMTP id 87B5735075ED; Sun, 14 Aug 2011 22:28:10 +0200 (CEST)
X-Virus-Scanned: amavisd-new at lu2-zimbra.skype.net
Received: from zimbra.skype.net ([127.0.0.1]) by localhost (zimbra.skype.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lxL5f-k3U5jp; Sun, 14 Aug 2011 22:28:09 +0200 (CEST)
Received: from zimbra.skype.net (lu2-zimbra.skype.net [78.141.177.82]) by zimbra.skype.net (Postfix) with ESMTP id 261273507635; Sun, 14 Aug 2011 22:28:09 +0200 (CEST)
Date: Sun, 14 Aug 2011 22:28:09 +0200 (CEST)
From: Koen Vos <koen.vos@skype.net>
To: Christian Hoene <hoene@uni-tuebingen.de>
Message-ID: <1586180885.47515.1313353689004.JavaMail.root@lu2-zimbra>
In-Reply-To: <000d01cc5ab9$30ef71f0$92ce55d0$@uni-tuebingen.de>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="----=_Part_47514_953282215.1313353689001"
X-Originating-IP: [67.203.191.242]
X-Mailer: Zimbra 6.0.9_GA_2686 (ZimbraWebClient - FF3.0 (Win)/6.0.9_GA_2686)
Cc: codec@ietf.org
Subject: Re: [codec] =?utf-8?q?T=C3=BCbingen_Test_plan?=
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 14 Aug 2011 20:27:30 -0000

Hi Christian, 

What audio bandwidths are you planning to use for the Opus test cases? 

You should probably call the SILK and Hybrid modes just "Voice" mode, and the CELT mode "Audio" mode. 

best, 
koen. 

----- Original Message -----
From: "Christian Hoene" <hoene@uni-tuebingen.de> 
To: codec@ietf.org 
Sent: Sunday, August 14, 2011 12:34:31 PM 
Subject: Re: [codec] Tübingen Test plan 




Hi, 



Thanks for the input. I have updated the test plan: 





Testplan for MUSHRA Listening Tests 
Focusing on a Conferencing Scenario 

Mansoor Hyder, Christian Hoene 
Universität Tübingen 
14. August 2011 
Version 2 Test setup 


MUSHRA tests following the recommendation ITU-R BS 1584.1 will take place on August 16th, from 9:00 till 19:00 in the office B303/B304 at the building Sand 13 in Tübingen, Germany. 

Tests will be done with Sennheiser HD 280 pro headphones, a PC and the software „RateIt“ version 0.1. 

Twenty participants have been selected from employees and students of the University of Tübingen. They have well to very good hearing abilities. Participant will receive soft drinks and/or coffee and 8 € compensation. 

Each session will last at most 60 minutes. This includes 15 to 20 minutes for explaining and practicing the tests, 25 to 30 minutes of testing, and 5 to 10 minutes of questioning and farewell. 

In total, 40 stimuli will be tested (in addition to the references). Test stimuli 


The focus of the stimuli is on a conferencing scenario, in which multiple persons speak at the same time. Only stereo test stimuli will be taken and only stereo coding modes will be tested. 

Five different reference samples will be taken: 

1. Female German voice (stereo) 

2. Two female voices (stereo, cross talking) 

3. One female voice (binaural rendered, moving) 

4. Two female voices (binaural rendered, cross talking) 

5. Music (an acappella song by “Die Prinzen”) 

The following codec and codec modes will be tests 

1. Opus, Silk mode, stereo, high complexity, 20ms frame size, at 16 kbps. 

2. Opus, Hybrid mode, stereo, high complexity, 20ms frame size, at 32 kbps 

3. Opus, Celt mode, high complexity, stereo, 20ms frame size at 64 kbps 

4. Opus, Celt mode, high complexity, stereo, 5ms frame size, at 96 kbps 

5. G.719, stereo, at 2x32 kbps (Fix-point-200806-Release-1.0a) 

6. AMR-WB+, stereo, at 16 kbps (Code from TS 26304 Version 6.6) 

7. AMR-WB+, stereo, at 32 kbps (Code from TS 26304 Version 6.6) 

8. TBD, stereo at 64 kbps, 20ms frame size 

In total, 5x8=40 stimuli are going to be tested. Script to make stimuli 




#!/bin/bash 

OPUS=../opus.testing/draft-ietf-codec-opus-07/test_opus 

G719e=../G.719-Fix-point-200806-Release-1.0a/out/encoder 

G719d=../G.719-Fix-point-200806-Release-1.0a/out/decoder 

AMRWBPe="wine ../26304_ANSI-C_source_code_v6_6_0/encoder" 

AMRWBPd="wine ../26304_ANSI-C_source_code_v6_6_0/decoder" 



I=1 

for file in ../reference\ items/*\.wav 

do 

echo $I $file 



sox "$file" -r48000 -c2 input.sw 

$OPUS 0 48000 2 16000 -cbr -framesize 20 -bandwidth WB input.sw output.sw >/dev/null 

sox -r48000 -c2 output.sw $I-opus.16k.wav 

$OPUS 0 48000 2 32000 -cbr -framesize 20 -bandwidth FB input.sw output.sw >/dev/null 

sox -r48000 -c2 output.sw $I-opus.32k.wav 

$OPUS 1 48000 2 64000 -cbr -framesize 20 -bandwidth FB input.sw output.sw >/dev/null 

sox -r48000 -c2 output.sw $I-opus.64k.wav 

$OPUS 1 48000 2 96000 -cbr -framesize 2.5 -bandwidth FB input.sw output.sw >/dev/null 

sox -r48000 -c2 output.sw $I-opus.96k.wav 

rm output.sw 



sox -r48000 -c2 input.sw -r48000 -c1 left.sw remix 1 0 

sox -r48000 -c2 input.sw -r48000 -c1 right.sw remix 0 1 

$G719e -r 32000 -i left.sw -o left.g192 

$G719e -r 32000 -i right.sw -o right.g192 

$G719d -i left.g192 -o outleft.sw 

$G719d -i right.g192 -o outright.sw 

sox -r48000 -c1 outleft.sw -c2 outleft.wav remix 1 0 

sox -r48000 -c1 outright.sw -c2 outright.wav remix 0 1 

sox -m outleft.wav outright.wav $I-g719.64k.wav 

rm left.sw right.sw left.g192 right.g192 outleft.sw outright.sw 

rm outleft.wav outright.wav input.sw 



$AMRWBPe -rate 16 -if "$file" -of out.wb+ 

$AMRWBPd -if out.wb+ -of $I-amrwbp.16k.wav 



$AMRWBPe -rate 32 -if "$file" -of out.wb+ 

$AMRWBPd -if out.wb+ -of $I-amrwbp.32k.wav 

rm out.wb+ 



sox "$file" -r48000 -c2 $I-reference.wav 

sox "$file" -r48000 -c2 $I-hidden.wav 

sox "$file" -r48000 -c2 $I-anchor.wav lowpass 3500 



let I=$I+1 

done 








--------------------------------------------------------------------------------- 

Dr.-Ing. Christian Hoene, University of Tübingen, Computer Science, Chair of 

Communication Networks, Research Group Interactive Communication Systems (ICS) 

Sand 13, 72076 Tübingen, Germany, Tel +49 7071 2970532, www.net.uni-tuebingen.de 






From: codec-bounces@ietf.org [mailto:codec-bounces@ietf.org] On Behalf Of Christian Hoene 
Sent: Sunday, August 14, 2011 3:22 PM 
To: codec@ietf.org 
Subject: [codec] Tübingen Test plan 



Hi, 



I was compiling the following tests plan for Tuesday. 

Any comments and hint to improve the test plan? 



With best regards, 



Christian 




_______________________________________________ 
codec mailing list 
codec@ietf.org 
https://www.ietf.org/mailman/listinfo/codec