Re: [ippm] Comparing Alternate Marking and Explicit Flow Measurements (Spin bit, ...)

"Lubashev, Igor" <> Sat, 27 March 2021 20:29 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 4CC253A112D; Sat, 27 Mar 2021 13:29:17 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.35
X-Spam-Status: No, score=-2.35 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.251, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id u8yWKOrFJ-2V; Sat, 27 Mar 2021 13:29:13 -0700 (PDT)
Received: from ( [IPv6:2620:100:9005:57f::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 8FC363A1128; Sat, 27 Mar 2021 13:29:12 -0700 (PDT)
Received: from pps.filterd ( []) by ( with SMTP id 12RKPXZB003210; Sat, 27 Mar 2021 20:29:08 GMT
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=jan2016.eng; bh=Inq32gKe402BubOGDK0bVMfpRMT9kc5NTak9tmxdEqI=; b=Hbet2TsbjnP/Ux0DBi5lbDSoQIS0/FfMzcmyBhXzfHPxzpktxbDgymvp93EHcsbZreGJ qSTYoJZPUhVqBYurASAlmmwuSe3RDaUEWl6V0KwcILPpDH0B6zcoaJbShuow4OMthjkU BUxBG0vZtnj27s0FHKMWxQC3UMxrufynBf7upxiqBsmGUb2JXYCRNPVV4sOn7JxvsCvt UyS44qi1Fo90bFcFxIqfl/PyCCWK71paUWcExwZBcNQd67hRESu6RAtNeEADlkdItiR/ ncBy4GeVs8/dcCP4Iol+7PEnBS6yP8+i6KR9VauXt2BQS+GYdlw2ZldjuFMRVtjxIqhA tA==
Received: from prod-mail-ppoint2 ( [] (may be forged)) by with ESMTP id 37hw85c09w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 27 Mar 2021 20:29:07 +0000
Received: from pps.filterd ( []) by ( with SMTP id 12RKKMhh013303; Sat, 27 Mar 2021 16:28:57 -0400
Received: from ([]) by with ESMTP id 37j01yhbjn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Sat, 27 Mar 2021 16:28:57 -0400
Received: from ( by ( with Microsoft SMTP Server (TLS) id 15.0.1497.2; Sat, 27 Mar 2021 16:28:57 -0400
Received: from ([]) by ([]) with mapi id 15.00.1497.012; Sat, 27 Mar 2021 16:28:57 -0400
From: "Lubashev, Igor" <>
To: Cociglio Mauro <>, "IETF IPPM WG (" <>, "" <>, Ian Swett <>, "" <>, "HAMCHAOUI Isabelle IMT/OLN" <>
CC: "" <>, "" <>
Thread-Topic: Comparing Alternate Marking and Explicit Flow Measurements (Spin bit, ...)
Thread-Index: AdcaadL89WUw5nTMSaO6t2YHkK4COgI2TAZA
Date: Sat, 27 Mar 2021 20:28:56 +0000
Message-ID: <>
References: <8f60ffc8e0fd4376ba911c03f5c43039@TELMBXD14BA020.telecomitalia.local>
In-Reply-To: <8f60ffc8e0fd4376ba911c03f5c43039@TELMBXD14BA020.telecomitalia.local>
Accept-Language: en-US
Content-Language: en-US
msip_labels: MSIP_Label_d6986fb0-3baa-42d2-89d5-89f9b25e6ac9_Enabled=true; MSIP_Label_d6986fb0-3baa-42d2-89d5-89f9b25e6ac9_SetDate=2021-03-16T13:47:41Z; MSIP_Label_d6986fb0-3baa-42d2-89d5-89f9b25e6ac9_Method=Standard; MSIP_Label_d6986fb0-3baa-42d2-89d5-89f9b25e6ac9_Name=Uso Interno; MSIP_Label_d6986fb0-3baa-42d2-89d5-89f9b25e6ac9_SiteId=6815f468-021c-48f2-a6b2-d65c8e979dfb; MSIP_Label_d6986fb0-3baa-42d2-89d5-89f9b25e6ac9_ActionId=50ed9e22-6867-47f8-99dd-40ec1c384e99; MSIP_Label_d6986fb0-3baa-42d2-89d5-89f9b25e6ac9_ContentBits=2
x-ms-exchange-transport-fromentityheader: Hosted
x-originating-ip: []
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-27_09:2021-03-26, 2021-03-27 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0 phishscore=0 malwarescore=0 mlxscore=0 bulkscore=0 suspectscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103270162
X-Proofpoint-GUID: _mhnEuHKDvyUOlK9mSklBCu-u6QfEzEt
X-Proofpoint-ORIG-GUID: _mhnEuHKDvyUOlK9mSklBCu-u6QfEzEt
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-03-27_09:2021-03-26, 2021-03-27 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 impostorscore=0 clxscore=1011 adultscore=0 malwarescore=0 priorityscore=1501 spamscore=0 mlxlogscore=999 lowpriorityscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103270163
X-Agari-Authentication-Results:; spf=${SPFResult} (sender IP is smtp.helo=prod-mail-ppoint2
Archived-At: <>
Subject: Re: [ippm] Comparing Alternate Marking and Explicit Flow Measurements (Spin bit, ...)
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF IP Performance Metrics Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Sat, 27 Mar 2021 20:29:17 -0000

Thank you, Mauro.

I would like to point out to IPPM WG that some of the proposed measurement techniques have already been implemented and had been running on the Internet in 2019 and 2020.  Namely, Akamai has implemented Loss Bits (L+Q) and enabled them on a large portion of QIUC traffic served to Orange Telecom users in several countries, while Orange implemented observers that collected and analyzed the measurements.  We have discussed the measurements and techniques in MAPRG during IETF-105 (and at other WGs and meetings).

Here is our IETF-105 MAPRG presentation discussing the data

In short, we found that unidirectional observations of QUIC traffic with L+Q bits alone to be effective for measuring both upstream and downstream packet loss and characterizing the magnitude of packet reordering.

During the last meeting Ian asked whether Akamai would implement and enable these measurements techniques for QUIC on the large scale, if all the relevant drafts get standardized by IETF.  The answer is that, yes, we would.  We are very interested in doing what it takes to improve Internet performance for the users, and we would work hard to implement techniques that can help network operators to improve QOS on their networks.

- Igor

> -----Original Message-----
> From: Cociglio Mauro <>
> Sent: Tuesday, March 16, 2021 9:48 AM
> To: IETF IPPM WG ( <>rg>;
> Cc:;
> Subject: [ippm] Comparing Alternate Marking and Explicit Flow
> Measurements (Spin bit, ...)
> Hi.
> Last Friday during the IPPM meeting,  after the "Explicit Flow Measurements"
> draft presentation
> (
> explicit-flow-measurements-01__;!!GjvTz_vk!CC-VjNQml5au1pXfCN50J-D-
> O99Zvrz686frIu4shWxPEipwQd7TTuOM3NTyNCU$ ), Greg Mirsky raised an
> issue that I think is very important.
> The question is about differences and similarities between the two types of
> production traffic packet marking for performance measurements, proposed
> in IETF and initiated in the IPPM WG: Alternate Marking and Explicit Flow
> Measurements.
> The first technique known as Alternate Marking or AM/PM (Alternate
> Marking Performance Monitoring) is defined, in general terms, in RFC8321
> (the point-to-point version) and RFC8889 (the multipoint version).
> It is essentially a Telco measurement, born to measure packet delay and loss
> between the input and output of a network, or between 2 internal points of
> the network, in order to identify and localize a problem. It is a network
> measure and it is the network operator that performs the marking by
> modifying packets on the fly.
> The strength of this technique comes from the decoupling of marking and
> measurement. We can mark all traffic, using a fixed marking interval (typically
> "big": from 1 second up to 5 minutes), then we decide what to measure
> based on the resources I want to use. In case of packet loss measurement
> we can start from a single packet counter for all traffic for each measurement
> point (possibility described in RFC8889), to have a network measurement, to
> arrive to a counter for each point-to-point connection you want to monitor
> (as described in RFC8321).
> In order to obtain the measurement it is necessary to compare the data
> collected from at least 2 measurement points (counters for packet loss,
> timestamps for delay). Then a "communication" between measurement
> points, or with a Network Measurement Center, is needed.
> There are already commercial implementations of this technique (also for
> IPv4) and IETF drafts that are standardizing it for various protocols (IPv6,
> MPLS, Segment Routing, BIER, ...).
> The Alternate Marking methodology is evolving into the draft "Big Data
> AltMark" (
> c2f-ippm-big-data-alt-mark-01__;!!GjvTz_vk!CC-VjNQml5au1pXfCN50J-D-
> O99Zvrz686frIu4shWxPEipwQd7TTuOM_yNIIgs$ ) that defines point-to-point
> flows measurements applying post processing to performance data collected
> by sampling a single network multipoint flow.
> The second marking technique for performance monitoring of packet
> networks has been called Explicit Flow Measurements (EFM), and is more
> recent because it's born with the Spin bit RTT measurement. And it came
> about primarily to have an end-to-end performance measure, from the
> terminal, on which an application is running, to the server at the opposite
> end of the network. EFM can be seen as complementary measures to
> Alternate Marking.
> It requires certain characteristics of the protocols to which it can be applied,
> which are client-server, and it is particularly convenient for protocols that
> prevent the marking of packets on the fly (e.g. QUIC), because the marking
> occurs only on the end-points of the connection.
> The disadvantage with respect to the previous technique is that it always
> works for client-server point-to-point connection, it is not possible to
> aggregate measurements saving on monitoring resources as described in
> RFC8889. The advantage is that it can also work with a single monitoring
> point, even if having more points enhances it and allows intradomain
> measurements. With a single measurement point you can obtain end-to-end
> measures (Spin bit, Delay bit for delay and Loss bit, rT loss bit for packet loss)
> or end-to-observer measures (sQuare bit and Reflection bit for packet loss).
> End-to-observer measurements and scalability considerations make it
> particularly convenient to place a measurement point on the client (see
> user-devices-explicit-monitoring-01__;!!GjvTz_vk!CC-VjNQml5au1pXfCN50J-
> D-O99Zvrz686frIu4shWxPEipwQd7TTuOMvZGbbWs$ ).
> Best Regards.
> Mauro
> _____________________
> Mauro Cociglio
> TIM - Telecom Italia
> Via G. Reiss Romoli, 274
> 10148 - Torino (Italy)
> Tel.: +390112285028
> Mobile: +393357669751
> _____________________
> TIM - Uso Interno - Tutti i diritti riservati.
> Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle
> persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla
> conoscenza di queste informazioni sono rigorosamente vietate. Qualora
> abbiate ricevuto questo documento per errore siete cortesemente pregati di
> darne immediata comunicazione al mittente e di provvedere alla sua
> distruzione, Grazie.
> This e-mail and any attachments is confidential and may contain privileged
> information intended for the addressee(s) only. Dissemination, copying,
> printing or use by anybody else is unauthorised. If you are not the intended
> recipient, please delete this message and any attachments and advise the
> sender by return e-mail, Thanks.
> Rispetta l'ambiente. Non stampare questa mail se non è necessario.
> _______________________________________________
> ippm mailing list
> _;!!GjvTz_vk!CC-VjNQml5au1pXfCN50J-D-
> O99Zvrz686frIu4shWxPEipwQd7TTuOMMXmeW4s$