Re: [video-codec] Performances, measurement of same (Re: The Can Has Landed)

"Mo Zanaty (mzanaty)" <mzanaty@cisco.com> Mon, 23 March 2015 14:55 UTC

Return-Path: <mzanaty@cisco.com>
X-Original-To: video-codec@ietfa.amsl.com
Delivered-To: video-codec@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B90101A8BC0 for <video-codec@ietfa.amsl.com>; Mon, 23 Mar 2015 07:55:47 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -14.511
X-Spam-Level:
X-Spam-Status: No, score=-14.511 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_DKIM_WL=-7.5] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O2Ydk0y3ihAK for <video-codec@ietfa.amsl.com>; Mon, 23 Mar 2015 07:55:46 -0700 (PDT)
Received: from alln-iport-2.cisco.com (alln-iport-2.cisco.com [173.37.142.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 542FE1A8BAF for <video-codec@ietf.org>; Mon, 23 Mar 2015 07:55:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=1376; q=dns/txt; s=iport; t=1427122546; x=1428332146; h=from:to:subject:date:message-id:references:in-reply-to: content-id:content-transfer-encoding:mime-version; bh=vZXsL/7930ED6m+2AeMvsWHJ2XVBBQJPk5IroAC/Bq8=; b=Xp1DYfMOGkgvb4C9+3nTk/3mKab190y2yrBKcwOb1SwTVuiSxGflitom 1WdNiJIeU76mgFcxPSIxEQNpsAN57Mn4CKAHQxMy2Jwn+Ky5OW2gNfGcU xjThMYlSPaE+S243zMGyV+pg46TDabGs2A+TIY1VJdZ0I9myOQOQi16Gj o=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A0AJBQA6KRBV/4QNJK1cgwaBLATMQAKBKUwBAQEBAQF9hBUBAQQ6OhUCAQg2EDIlAgQBEogvyAwBAQEBAQEBAwEBAQEBAQEbiyGEfIQtAQSQT4lulCoig25vgUR/AQEB
X-IronPort-AV: E=Sophos;i="5.11,452,1422921600"; d="scan'208";a="134587742"
Received: from alln-core-10.cisco.com ([173.36.13.132]) by alln-iport-2.cisco.com with ESMTP; 23 Mar 2015 14:55:45 +0000
Received: from xhc-aln-x12.cisco.com (xhc-aln-x12.cisco.com [173.36.12.86]) by alln-core-10.cisco.com (8.14.5/8.14.5) with ESMTP id t2NEtjcT016784 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Mon, 23 Mar 2015 14:55:45 GMT
Received: from xmb-rcd-x14.cisco.com ([169.254.4.61]) by xhc-aln-x12.cisco.com ([173.36.12.86]) with mapi id 14.03.0195.001; Mon, 23 Mar 2015 09:55:45 -0500
From: "Mo Zanaty (mzanaty)" <mzanaty@cisco.com>
To: Harald Alvestrand <harald@alvestrand.no>, "video-codec@ietf.org" <video-codec@ietf.org>
Thread-Topic: [video-codec] Performances, measurement of same (Re: The Can Has Landed)
Thread-Index: AQHQZXlwNdqDcNEL0E6Z9vBFpmqNQw==
Date: Mon, 23 Mar 2015 14:55:44 +0000
Message-ID: <D1359BB4.4A2D5%mzanaty@cisco.com>
References: <CACrD=+_D+psUeWevMuwp0bnxqdcJpo3Zo3Og4E6kkGH1uuzxdA@mail.gmail.com> <CAMzhQmNymkEMgbw-gUGhKEgCYh1yXo8MRkP-8FNfQm8tVNhzbA@mail.gmail.com> <550B6CCC.70706@mozilla.com> <551009A9.1010108@alvestrand.no>
In-Reply-To: <551009A9.1010108@alvestrand.no>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.4.8.150116
x-originating-ip: [64.100.32.216]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <355F548D72B7E34A8155F44DDB02E167@emea.cisco.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Archived-At: <http://mailarchive.ietf.org/arch/msg/video-codec/fwea1BYPP1KUT5q_pvKyW7YHzB0>
Subject: Re: [video-codec] Performances, measurement of same (Re: The Can Has Landed)
X-BeenThere: video-codec@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Video codec BoF discussion list <video-codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/video-codec>, <mailto:video-codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/video-codec/>
List-Post: <mailto:video-codec@ietf.org>
List-Help: <mailto:video-codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/video-codec>, <mailto:video-codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Mar 2015 14:55:47 -0000

Complexity (or more broadly, resource requirements, which encompass
aspects beyond compute cycles like memory and memory bandwidth) should be
a fundamental part of the evaluation methodology. But there are many
pitfalls to avoid here.

Intrinsic algorithmic complexity is difficult to discern with any
precision, since any metric necessarily depends on the level of
implementation optimization (from simple loop unrolling to vector
intrinsics or full assembly).

Optimization can hamper experimentation, so it is usually not a good idea
in the research stage.

Configuration points are also important to consider. Having configuration
points that are impractical for real-world use is still valuable for
steering the research. This should not be the primary evaluation criteria,
which should attempt to focus on real-world use. But such configuration
points should be allowed and analyzed.

Mo


On 3/23/15, 8:40 AM, Harald Alvestrand <harald@alvestrand.no> wrote:
I also wonder a bit about how we should treat evaluation of performance
- a traditional video codec development technique has been to develop a
codec that takes hours-per-second to encode a video, and then seek to
optimize it by a factor of a hundred before shipping it. But sometimes,
there's just limits to how much one can achieve.
How should we treat that aspect in evaluation?