Re: [ippm] [Raw] New Version Notification for draft-ietf-raw-oam-support-01.txt

["Pascal Thubert (pthubert)" <[e-mail]>]

Hello again Greg:

I read I-D.mirsky-ippm-hybrid-two-step as part of a discussion at RAW, I really liked it.

There might be a race condition with the HTS Follow-up Timer in the draft whereby multiple nodes along the path would time out at the almost the same time and generate a follow-up message, leading to a cascade effect. 

Seems to me that the Timer should be augmented along the path to cover the jitter of a follow-up packet generated upstream.

This could be done using the number of hops times a large enough jitter value at each hop. Note that the jitter of the Trigger might be lower than that of the follow up.

To enable that you could for instance signal the hop count in the Trigger message. The hop count could be indicated when the Follow-up is regenerated due to MTU issue so we can order the multiple messages at the receiver and eliminate duplicates due to a late follow up.
 
Note finally that routers usually have jitter in their timers to avoid network sync effects; we do not want such jitter here.

Keep safe;

Pascal
 
> > -----Original Message-----
> > From: [e-mail] <[e-mail]>
> > Sent: jeudi 3 juin 2021 23:37
> > To: Pascal Thubert (pthubert) <[e-mail]>
> > Cc: [e-mail]; [e-mail];
> > [e-mail]; [e-mail]; [e-mail]
> > Subject: Re:[Raw] New Version Notification for draft-ietf-raw-oam-
> > support-01.txt
> >
> > Hi Pascal,
> > thank you for your comments and questions. We've been addressing
> > comments on OAM Framework for DetNet and it might be that one of the
> > updates in the latest version be of use in addressing your question on
> > the teleemtry
> > collection:
> > How we observe is not discussed yet. I’d like to wrote text about data
> > collection packets (bees) that travel the reverse path and gather the
> > latest metrics (e.g. DLEP) at each hop. Also I’d like to see text on
> > IPPM, and whatever you guys think is best suited. The goal of the
> > architecture is to give a blueprint/skeleton/direction for the
> > solution drafts later.
> >
> > Below is the quote from draft-ietf-detnet-oam-framework:
> >   Also, we can characterize methods of transporting OAM information
> > relative to the path of data.  For instance, OAM information may be
> > transported in-band or out-of-band with the data flow.  In case of   the
> > former, the telemetry information uses resources allocated for   the
> > monitored DetNet flow.  If an in-band method of transporting   telemetry
> > is used, the amount of generated information needs to be   carefully
> > analyzed, and additional resources must be reserved.   [I-D.ietf-ippm-
> > ioam-data] defines the in-band transport mechanism   where telemetry
> > information is collected in the data packet on which   information is
> > generated.  Two tracing methods are described - end-   to-end, i.e., from
> > the ingress and egress nodes, and hop-by-hop,   i.e., like end-to-end
> > with additional information from transit nodes.   [I-D.ietf-ippm-ioam-
> > direct-export] and   [I-D.mirsky-ippm-hybrid-two-step] are examples of
> > out-of-band telemetry transport.  In the former case, information is tra
> >  nsported   by each node traversed by the data packet of the monitored
> > DetNet   flow in a specially constructed packet.  In the latter,
> > information   is collected in a sequence of follow-up packets that
> > traverse the   same path as the data packet of the monitored DetNet flow.
> > In both   methods, transport of the telemetry can avoid using resources
> > allocated for the DetNet domain.
> >
> > We always welcome comments and greatly appreciate your suggestions.
> >
> > Regards,
> > Greg Mirsky
> > Sr. Standardization Expert
> > 预研标准部/有线研究院/有线产品经营部  Standard Preresearch Dept./Wireline
> Product R&D
> > Institute/Wireline Product Operation Division
> >
> > E: [e-mail]
> > www.zte.com.cn
> > ------------------Original Mail------------------
> > Sender: PascalThubert(pthubert)
> > To: Georgios PAPADOPOULOS;CARLOS JESUS BERNARDOS CANO;
> > CC: Fabrice Theoleyre;[e-mail];Xavi Vilajosana Guillen;
> > Date: 2021/06/03 05:01
> > Subject: Re: [Raw] New Version Notification for draft-ietf-raw-oam-
> > support-01.txt
> > --
> > RAW mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/raw
> >
> > Hello All:
> >
> > I’d be happy to add text in the raw architecture document (section
> > 4.3) on the high level solution space that we want to consider.
> > Right now we say what we observe, all segments (in a mesh) or just the
> > wireless access (that’s supposed to be the lossy part and that pays
> > for the rest of the path anyway).
> >
> > We also have a discussion on how we tag the packets (with a HbH with
> > is consistent with 6TiSCH and the current directions at 6MAN,
> > https://datatracker.ietf.org/doc/draft-hinden-6man-hbh-processing/, or
> > SRv6 SRH) to identify the flows. In any fashion, this allows to use
> > the same identification  for the OAM packets and the flow itself.
> >
> > How we observe is not discussed yet. I’d like to wrote text about data
> > collection packets (bees) that travel the reverse path and gather the
> > latest metrics (e.g. DLEP) at each  hop. Also I’d like to see text on
> > IPPM, and whatever you guys think is best suited. The goal of the
> > architecture is to give a blueprint/skeleton/direction for the
> > solution drafts later.
> >
> > Any suggestions?
> >
> > Pascal
> >
> > From: RAW <[e-mail]> On Behalf Of Georgios PAPADOPOULOS
> > Sent: jeudi 3 juin 2021 11:21
> > To: CARLOS JESUS BERNARDOS CANO <[e-mail]>
> > Cc: Fabrice Theoleyre <[e-mail]>; [e-mail]; Xavi
> > Vilajosana Guillen <[e-mail]>
> > Subject: Re: [Raw] New Version Notification for draft-ietf-raw-oam-
> > support-01.txt
> >
> > Hello all,
> >
> >
> >
> > On 27 May 2021, at 11:41, CARLOS JESUS BERNARDOS CANO
> > <[e-mail]>
> > wrote:
> >
> > Hi Xavi, Fabrice, all,
> >
> > Thanks Xavi for all the comments and Fabrice for taking care of them.
> >
> > On the requirements/solutions point, my personal opinion is that
> > solutions should be addressed in a different draft, when charter
> > allows so (not sure we can work on solutions  now).
> >
> > +1,
> > Georgios
> >
> >
> >
> > Thanks!
> >
> > Carlos
> >
> > On Thu, May 27, 2021 at 11:00 AM Fabrice Theoleyre
> > <[e-mail]>
> > wrote:
> > Dear Xavi,
> >
> > Thank you for your comments!
> > I will try to reply inline.
> >
> >
> > RAW                                                         F. Theoleyre
> > Internet-Draft                                                      CNRS
> > Updates: draft-ietf-raw-oam-support-00                   G. Papadopoulos
> > (if approved)                                    IMT Atlantique
> > Intended status: Informational                                 G. Mirsky
> > Expires: November 25, 2021                                     ZTE Corp.
> > CJ. Bernardos
> > UC3M
> > May 24, 2021
> >
> >
> > Operations, Administration and Maintenance (OAM) features for RAW
> > draft-ietf-raw-oam-support-01
> >
> > Abstract
> >
> > Some critical applications may use a wireless infrastructure.
> > However, wireless networks exhibit a bandwidth of several orders of
> > magnitude lower than wired networks.  Besides, wireless transmissions
> > are lossy by nature; the probability that a packet cannot be decoded
> > correctly by the receiver may be quite high.
> > XV>> the phrase that follows is vague. I would stress what the
> > XV>> challenges
> > are.
> > In these conditions,
> > guaranteeing that the network infrastructure works properly is
> > particularly challenging, since we need to address some issues
> > specific to wireless networks.
> > FT> Thank you, I updated this part: "In these conditions, providing
> > FT> high
> > reliability and a low delay is challenging."
> >
> > This document lists the requirements
> > of the Operation, Administration, and Maintenance (OAM) features
> > recommended to construct a predictable communication infrastructure on
> > top of a collection of wireless segments.
> > XV>> Only requirements? are we going to propose a model/mechanism?
> >
> > FT> to my mind, providing mechanisms seem out of scope for this draft
> > FT> (up
> > to now). I will create a separated thread to discuss this problem,
> > since I guess it deserves a discussion  with the WG.
> >
> >
> >
> > This document describes
> > the benefits, problems, and trade-offs for using OAM in wireless
> > networks to achieve Service Level Objectives (SLO).
> >
> > Status of This Memo
> >
> > This Internet-Draft is submitted in full conformance with the
> > provisions of BCP 78 and BCP 79.
> >
> > Internet-Drafts are working documents of the Internet Engineering Task
> > Force (IETF).  Note that other groups may also distribute working
> > documents as Internet-Drafts.  The list of current Internet- Drafts is
> > at https://datatracker.ietf.org/drafts/current/.
> >
> > Internet-Drafts are draft documents valid for a maximum of six months
> > and may be updated, replaced, or obsoleted by other documents at any time.
> > It is inappropriate to use Internet-Drafts as reference material or to
> > cite them other than as "work in progress."
> >
> > This Internet-Draft will expire on November 25, 2021.
> >
> >
> >
> >
> >
> > Theoleyre, et al.       Expires November 25, 2021               [Page 1]
> >
> > Internet-Draft            OAM features for RAW                  May 2021
> >
> >
> > Copyright Notice
> >
> > Copyright (c) 2021 IETF Trust and the persons identified as the
> > document authors.  All rights reserved.
> >
> > This document is subject to BCP 78 and the IETF Trust's Legal
> > Provisions Relating to IETF Documents
> > (https://trustee.ietf.org/license-info) in effect on the date of
> > publication of this document.  Please review these documents
> > carefully, as they describe your rights and restrictions with respect
> > to this document.  Code Components extracted from this document must
> > include Simplified BSD License text as described in Section 4.e of the
> > Trust Legal Provisions and are provided without warranty as described
> > in the Simplified BSD License.
> >
> > Table of Contents
> >
> > 1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
> > 1.1.  Terminology . . . . . . . . . . . . . . . . . . . . . . .   4
> > 1.2.  Acronyms  . . . . . . . . . . . . . . . . . . . . . . . .   5
> > 1.3.  Requirements Language . . . . . . . . . . . . . . . . . .   6
> > 2.  Role of OAM in RAW  . . . . . . . . . . . . . . . . . . . . .   6
> > 2.1.  Link concept and quality  . . . . . . . . . . . . . . . .   7
> > 2.2.  Broadcast Transmissions . . . . . . . . . . . . . . . . .   7
> > 2.3.  Complex Layer 2 Forwarding  . . . . . . . . . . . . . . .   8
> > 2.4.  End-to-end delay  . . . . . . . . . . . . . . . . . . . .   8
> > 3.  Operation . . . . . . . . . . . . . . . . . . . . . . . . . .   8
> > 3.1.  Information Collection  . . . . . . . . . . . . . . . . .   8
> > 3.2.  Continuity Check  . . . . . . . . . . . . . . . . . . . .   9
> > 3.3.  Connectivity Verification . . . . . . . . . . . . . . . .   9
> > 3.4.  Route Tracing . . . . . . . . . . . . . . . . . . . . . .   9
> > 3.5.  Fault Verification/detection  . . . . . . . . . . . . . .   9
> > 3.6.  Fault Isolation/identification  . . . . . . . . . . . . .  10 4.
> > Administration  . . . . . . . . . . . . . . . . . . . . . . .  10 4.1.
> > Worst-case metrics  . . . . . . . . . . . . . . . . . . .  11 4.2.
> > Efficient data retrieval  . . . . . . . . . . . . . . . .  11 4.3.
> > Reporting OAM packets to the source . . . . . . . . . . .  12 5.
> > Maintenance . . . . . . . . . . . . . . . . . . . . . . . . .  12 5.1.
> > Soft transition after reconfiguration . . . . . . . . . .  12 5.2.
> > Predictive maintenance  . . . . . . . . . . . . . . . . .  12 6.  IANA
> > Considerations . . . . . . . . . . . . . . . . . . . . .  13 7.
> > Security Considerations . . . . . . . . . . . . . . . . . . .  13 8.
> > Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .  13 9.
> > Informative References  . . . . . . . . . . . . . . . . . . .  13
> > Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  14
> >
> >
> >
> >
> >
> >
> > Theoleyre, et al.       Expires November 25, 2021               [Page 2]
> >
> > Internet-Draft            OAM features for RAW                  May 2021
> >
> >
> > 1.  Introduction
> >
> > Reliable and Available Wireless (RAW) is an effort that extends DetNet
> > to approach end-to-end deterministic performances over a network that
> > includes
> > XV>> Only scheduled? can we generalize to non-scheduled? e.g WiFi is
> > XV>> not
> > scheduled.
> > scheduled wireless segments.  In wired networks, many approaches try
> > to enable Quality of Service (QoS) by implementing traffic
> > differentiation so that routers handle each type of packets
> > differently.  However, this differentiated treatment was expensive for
> > most applications.
> >
> > Deterministic Networking (DetNet) [RFC8655] has proposed to provide a
> > bounded end-to-end latency on top of the network infrastructure,
> > comprising both Layer 2 bridged and Layer 3 routed segments.  Their
> > work encompasses the data plane, OAM, time synchronization,
> > management, control, and security aspects.
> >
> > However, wireless networks create specific challenges.  First of all,
> > radio bandwidth is significantly lower than for wired networks.  In
> > these conditions, the volume of signaling messages has to be very limited.
> > Even worse, wireless links are lossy: a Layer 2 transmission may or
> > may not be decoded correctly by the receiver, depending on a broad set
> > of parameters.  Thus, providing high reliability through wireless
> > segments is particularly challenging.
> >
> > Wired networks rely on the concept of _links_. All the devices
> > attached to a link receive any transmission.  The concept of a link in
> > wireless networks is somewhat different from what many are used to in
> > wireline networks.  A receiver may or may not receive a transmission,
> > depending on the presence of a colliding transmission, the radio
> > channel's quality, and the external interference.  Besides, a wireless
> > transmission is broadcast by nature: any _neighboring_ device may be able
> to decode it.
> > XV>> This document instead of The document?
> > FT> thank you!
> >
> >
> >
> > The document includes detailed
> > information on what the implications for the OAM features are.
> >
> > Last but not least, radio links present volatile characteristics.  If
> > the wireless networks use an unlicensed band, packet losses are not
> > anymore temporally and spatially independent.  Typically, links may
> > exhibit a very bursty characteristic, where several consecutive
> > packets may be dropped.  Thus, providing availability and reliability
> > on top of the wireless infrastructure requires specific Layer 3
> > mechanisms to counteract these
> > XV>> unpredictable instead of bursty?
> > bursty losses.
> >
> > FT> I make here a distinction: i) unpredictable=we don’t know, ii)
> > bursty=the radio channel changes for a small duration (e.g., a few
> > packets), for instance because of external  interference.
> > FT> I guess that both problems are of interest but are dissimilar. I
> > reformulated into: "Typically, links may exhibit a very bursty
> > characteristic, where several consecutive packets  may be dropped,
> > because of e.g. temporary external interference.”
> >
> >
> >
> >
> > Operations, Administration, and Maintenance (OAM) Tools are of primary
> > importance for IP networks [RFC7276].
> > XV>> They define
> > It defines a toolset
> > for fault detection, isolation, and performance measurement.
> > FT> thank you!
> >
> >
> >
> > The primary purpose of this document is to detail the specific
> > requirements of the OAM features recommended to construct a
> >
> >
> >
> > Theoleyre, et al.       Expires November 25, 2021               [Page 3]
> >
> > Internet-Draft            OAM features for RAW                  May 2021
> >
> >
> > predictable communication infrastructure on top of a collection of
> > wireless segments.  This document describes the benefits, problems,
> > and trade-offs for using OAM in wireless networks to provide
> > availability and predictability.
> >
> > 1.1.  Terminology
> >
> > In this document, the term OAM will be used according to its
> > definition specified in [RFC6291].  We expect to implement an OAM
> > framework in RAW networks to maintain a real-time view of the network
> > infrastructure, and its ability to respect the Service Level
> > Objectives (SLO), such as delay and reliability, assigned to each data
> flow.
> >
> > We re-use here the same terminology as [detnet-oam]:
> >
> > o  OAM entity: a data flow to be monitored for defects and/or its
> > performance metrics measured.;  XV>> remove the '.'
> >
> > o  Maintenance End Point (MEP): OAM devices crossed when entering/
> > exiting the network.  In RAW, it corresponds mostly to the source or
> > destination of a data flow.  OAM message can be exchanged between two
> > MEPs;
> >
> > o  Maintenance Intermediate endPoint (MIP): an OAM system along the
> > flow; a MIP MAY respond to an OAM message generated by the MEP;
> >
> > o  control/management/data plane: the control and management planes
> > are used to configure and control the network (long-term).  The data
> > plane takes the individual decision.  Relative to a data flow, the
> > control and/or management plane can be out-of-band;
> >
> > o  Active measurement methods (as defined in [RFC7799]) modify a
> > normal data flow by inserting novel fields, injecting specially
> > constructed test packets [RFC2544]).  It is critical for the quality
> > of information obtained using an active method that generated test
> > packets are in-band with the monitored data flow.
> > In other words, a test packet is required to cross the same network
> > nodes and links and receive the same Quality of Service
> > (QoS) treatment as a data packet.  Active methods may implement one of
> > these two strategies:
> >
> > *  In-band: control information follows the same path as the data
> > packets.  In other words, a failure in the data plane may prevent the
> > control information to reach the destination (e.g., end-device or
> > controller).
> >
> >
> >
> >
> >
> > Theoleyre, et al.       Expires November 25, 2021               [Page 4]
> >
> > Internet-Draft            OAM features for RAW                  May 2021
> >
> >
> > *  out-of-band: control information is sent separately from the data
> > packets.  Thus, the behavior of control vs. data packets may differ;
> >
> > o  Passive measurement methods [RFC7799] infer information by
> > observing unmodified existing flows.
> >
> > We also adopt the following terminology, which is particularly
> > relevant for RAW segments.
> >
> > o  piggybacking vs. dedicated control packets: control information may
> > be encapsulated in specific (dedicated) control packets.
> > Alternatively, it may be piggybacked in existing data packets, when
> > the MTU is larger than the actual packet length.
> > Piggybacking makes specifically sense in wireless networks, as the
> > cost (bandwidth and energy) is not linear with the packet size.
> >
> > o  router-over vs. mesh under: a control packet is either forwarded
> > directly to the layer-3 next hop (mesh under) or handled hop-by- hop
> > by each router.  While the latter option consumes more resources, it
> > allows to collect additionnal intermediary information, particularly
> > relevant in wireless networks.
> >
> > o  Defect: a temporary change in the network (e.g., a radio link which
> > is broken due to a mobile obstacle);
> >
> > o  Fault: a definite change which may affect the network performance,
> > e.g., a node runs out of energy.
> >
> > o  End-to-end delay: the time between the packet generation and its
> > reception by the destination.
> >
> > 1.2.  Acronyms
> >
> > OAM Operations, Administration, and Maintenance
> >
> > DetNet Deterministic Networking
> >
> > PSE Path Selection Engine [I-D.pthubert-raw-architecture]
> >
> > QoS Quality of Service
> >
> > RAW Reliable and Available Wireless
> >
> > SLO Service Level Objective
> >
> > SNMP Simple Network Management Protocol
> >
> >
> >
> >
> > Theoleyre, et al.       Expires November 25, 2021               [Page 5]
> >
> > Internet-Draft            OAM features for RAW                  May 2021
> >
> >
> > SDN Software-Defined Network
> >
> > 1.3.  Requirements Language
> >
> > The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
> > "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
> > "OPTIONAL" in this document are to be interpreted as described in BCP
> > 14 [RFC2119] [RFC8174] when, and only when, they appear in all
> > capitals, as shown here.
> >
> > 2.  Role of OAM in RAW
> >
> > RAW networks expect to make the communications reliable and
> > predictable on top of a wireless network infrastructure.  Most
> > critical applications will define an SLO to be required for the data
> > flows it generates.  RAW considers network plane protocol elements
> > such as OAM to improve the RAW operation at the service and the forwarding
> sub-layers.
> >
> > To respect strict guarantees, RAW relies on the Path Selection Engine
> > (PSE) (as defined in [I-D.pthubert-raw-architecture] to monitor and
> > maintain the network.
> > XV>> To me, the example is not clear. We need to differentiate between
> > the L2 scheduler or any other access mechanisms (Can be CSMA/CA?) to
> > the PSE mechanism, which runs at L3 and cannot change the operation of
> > the L2 scheduler but react according to the  path characteristics by
> > introducing redundancy or selecting other routes to mitigate the
> > effect of an underperforming path.
> > As an example, a Software-Defined Network
> > (SDN) controller may be used to schedule the transmissions in the
> > deployed network, based on the radio link characteristics, SLO of the
> > flows, the number of packets to forward.  Thus, resources have to be
> > provisioned a priori to handle any defect.
> >
> > FT> thank you. I reformulate the whole paragraph to make a distinction
> > FT> between L2 and L3: "To respect strict guarantees, RAW relies on
> > FT> the Path Selection Engine (PSE) (as
> > defined in <xref target="I-D.pthubert-raw-architecture" /> to monitor
> > and maintain the L3 network. A L2 scheduler may be used to allocate
> > transmission opportunities, based on the radio link characteristics,
> > SLO of the flows, the number of packets to forward.
> > The PSE exploits the L2 ressources reserved by the scheduler, and
> > organizes the L3 paths to introduce redundancy, fault tolerance, and
> > create backup paths."
> >
> > XV>> OAM, should be monitoring that and provision alternate routes or
> > request to the underlying layer to provision support to more capacity
> > to create an overprovisioning effect.
> > OAM represents the core
> > of the pre-provisioning process and maintains the network operational
> > by updating the schedule dynamically.
> >
> > FT> I also reformulated this part: "OAM represents the core of the
> > FT> pre-
> > provisioning process by supervising the network.
> > It maintains maintains a global view of the network resources, to
> > detect defects, faults, over-provisionning, anomalies.
> >
> >
> > "
> > Fault-tolerance also assumes that multiple paths have to be
> > provisioned so that an end-to-end circuit keeps on existing whatever the
> conditions.
> > The Packet Replication and Elimination Function
> > ([PREF-draft]) on a node is typically controlled by the PSE.  OAM
> > mechanisms can be used to monitor that PREOF is working correctly on a
> > node and within the domain.
> >
> > To be energy-efficient, reserving some dedicated out-of-band resources
> > for OAM seems idealistic, and only in-band solutions are considered here.
> >
> > RAW supports both proactive and on-demand troubleshooting.
> >
> > XV>> Should we extend this concept? I guess this is key.
> > FT> thank you. I developed this part. I hope the novel version is more
> > precise: "RAW supports both proactive and on-demand troubleshooting.
> > Proactively, it is necessary to detect anomalies, to report defects,
> > or to reduce over-provisionning if it is not required.
> > However, on-demand may also be required to identify the cause of a
> > specific defect.
> > Indeed, some specific faults may only be detected with a global,
> > detailed view of the network, which is too expensive to acquire in the
> > normal operating mode."
> >
> > FT> for the specific solutions to apply, I guess that they should fit
> > FT> in
> > another draft.
> >
> >
> >
> > The specific characteristics of RAW are discussed below.
> >
> >
> >
> >
> >
> >
> >
> >
> > Theoleyre, et al.       Expires November 25, 2021               [Page 6]
> >
> > Internet-Draft            OAM features for RAW                  May 2021
> >
> >
> > 2.1.  Link concept and quality
> >
> > In wireless networks, a _link_ does not exist physically.  A device
> > has a set of *neighbors* that correspond to all the devices that have
> > a non null probability of receiving correctly its packets.  We make a
> > distinction between:
> >
> > o  point-to-point (p2p) link with one transmitter and one receiver.
> > These links are used to transmit unicast packets.
> >
> > o  point-to-multipoint (p2m) link associates one transmitter and a
> > collection of receivers.  For instance, broadcast packets assume the
> > existence of p2m links to avoid duplicating a broadcast packet to
> > reach each possible radio neighbor.
> > XV>> maybe we need to clarify the concept of scheduled radio link.
> > XV>> Would
> > that also work for WiFi?
> >
> > FT> to my mind, yes. For instance WiFi6 uses the TWT concept (target
> > wake-up time). And it has been extended in version 6 (compared with
> > ah) typically to include the p2m case  (broadcast actually).
> > FT> isn’t it the role of the technology draft?
> > FT> BTW, I also reformulated the sentence: "In scheduled radio
> > FT> networks,
> > p2m and p2p links are commonly not scheduled simultaneously to save
> > energy, and/or to reduce the number  of collisions."
> >
> >
> >
> > In scheduled radio networks, p2m and p2p links are commonly not
> > scheduled simultaneously to save energy.  More precisely, only one
> > part of the neighbors may wake-up at a given instant.
> >
> > Anycast are used in p2m links to improve the reliability.  A
> > collection of receivers are scheduled to wake-up simutaneously, so
> > that the transmission fails only if none of the receivers is able to
> > decode the packet.
> >
> > Each wireless link is associated with a link quality, often measured
> > as the Packet Delivery Ratio (PDR), i.e., the probability that the
> > receiver can decode the packet correctly.  It is worth noting that
> > this link quality depends on many criteria, such as the level of
> > external interference, the presence of concurrent transmissions, or
> > the radio channel state.  This link quality is even time-variant.
> > For p2m links, we have consequently a collection of PDR (one value per
> > receiver).  Other more sophisticated, aggregated metrics exist for
> > these p2m links, such as [anycast-property]
> >
> > 2.2.  Broadcast Transmissions
> >
> > In modern switching networks, the unicast transmission is delivered
> > uniquely to the destination.  Wireless networks are much closer to the
> > ancient *shared access* networks.  Practically, unicast and broadcast
> > frames are handled similarly at the physical layer.  The link layer is
> > just in charge of filtering the frames to discard irrelevant
> > receptions (e.g., different unicast MAC address).
> >
> > However, contrary to wired networks, we cannot be sure that a packet
> > is received by *all* the devices attached to the Layer 2 segment.  It
> > depends on the radio channel state between the transmitter(s) and the
> > receiver(s).  In particular, concurrent transmissions may be possible
> > or not, depending on the radio conditions (e.g., do the different
> >
> >
> >
> > Theoleyre, et al.       Expires November 25, 2021               [Page 7]
> >
> > Internet-Draft            OAM features for RAW                  May 2021
> >
> >
> > transmitters use a different radio channel or are they sufficiently
> > spatially separated?)
> >
> > 2.3.  Complex Layer 2 Forwarding
> >
> > Multiple neighbors may receive a transmission.  Thus, anycast Layer 2
> > forwarding helps to maximize the reliability by assigning multiple
> > receivers to a single transmission.  That way, the packet is lost only
> > if
> > *none* of the receivers decode it.  Practically, it has been proven
> > that different neighbors may exhibit very different radio conditions,
> > and that reception independency may hold for some of them [anycast-
> property].
> >
> > 2.4.  End-to-end delay
> >
> > In a wireless network, additionnal transmissions opportunities are
> > provisionned to accomodate packet losses.  Thus, the end-to-end delay
> > consists of:
> >
> > o  Transmission delay, which is fixed and depends mainly on the data
> > rate, and the presence or absence of an acknowledgement.
> >
> > o  Residence time, corresponds to the buffering delay and depends on
> > the schedule.  To account for retransmisisons, the residence time is
> > equal to the difference between the time of last reception from the
> > previous hop (among all the retransmisions) and the time of emission
> > of the last retransmission.
> >
> > 3.  Operation
> >
> > OAM features will enable RAW with robust operation both for forwarding
> > and routing purposes.
> >
> > 3.1.  Information Collection
> >
> > The model to exchange information should be the same as for DetNet
> > network, for the sake of inter-operability.  YANG may typically
> > fulfill this objective.
> >
> > However, RAW networks imply specific constraints (e.g., low bandwidth,
> > packet losses, cost of medium access) that may require to minimize the
> > volume of information to collect.  Thus, we discuss in Section 4.2
> > different ways to collect information, i.e., transfer physically the
> > OAM information from the emitter to the receiver.
> >
> >
> >
> >
> >
> >
> >
> > Theoleyre, et al.       Expires November 25, 2021               [Page 8]
> >
> > Internet-Draft            OAM features for RAW                  May 2021
> >
> >
> > 3.2.  Continuity Check
> >
> > Similarly to DetNet, we need to verify that the source and the
> > destination are connected (at least one valid path exists)
> >
> > 3.3.  Connectivity Verification
> >
> > As in DetNet, we have to verify the absence of misconnection.  We
> > focus here on the RAW specificities.
> >
> > Because of radio transmissions' broadcast nature, several receivers
> > may be active at the same time to enable anycast Layer 2 forwarding.
> > Thus, the connectivity verification must test any combination.  We
> > also consider priority-based mechanisms for anycast forwarding, i.e.,
> > all the receivers have different probabilities of forwarding a packet.
> > To verify a delay SLO for a given flow, we must also consider all the
> > possible combinations, leading to a probability distribution function
> > for end-to- end transmissions.  If this verification is implemented
> > naively, the number of combinations to test may be exponential and too
> > costly for wireless networks with low bandwidth.
> >
> > 3.4.  Route Tracing
> >
> > XV>> the following statement is not completely true. There are some
> > XV>> networks that are meshed. but most of wireless are point to point
> > XV>> or hub and spoke. (e.g wifi, ble, lorawan, lte, etc...)
> >
> > FT> even in WiFi, we may have inter-BSS transmissions. Similarly, we
> > FT> may
> > have several AP that desserve a STA. But I agree, it has to be
> > clarified, and ”meshed” is probably not  the right term. "Wireless
> > networks are broadcast by nature: a radio transmission can be decoded
> > by any radio neighbor.
> > In multihop wireless networks, several paths exist between two endpoints.
> > In hub networks, a device may be covered by several Access Points.
> > We should choose the most efficient path or AP, concerning
> > specifically the reliability, and the delay.”
> >
> >
> >
> > Wireless networks are meshed by nature: we have many redundant radio
> > links.  These meshed networks are both an asset and a drawback: while
> > several paths exist between two endpoints, and we should choose the
> > most efficient one(s), concerning specifically the reliability, and the
> delay.
> > XV>> Note that the statements in the previous paragraph also apply to
> > non-meshed networks, in a WiFi deployment you can have multiple APs so
> > there can be potentially multiple receivers for an emitting node.
> >
> > FT> exactly what I had in mind :-) I reformulated also this paragraph:
> > "Thus, multipath routing / multi-attachment can be considered to make
> > the network fault-tolerant.
> > Even better, we can exploit the broadcast nature of wireless networks
> > to
> > exploit: we may have multiple Maintenance Intermediate Endpoints (MIP)
> > for each of this kind  of hop.
> > While it may be reasonable in the multi-attachment case, the
> > complexity quickly increases with the path length.
> > Indeed, each Maintenance Intermediate Endpoint has several possible
> > next hops in the forwarding plane.
> > Thus, all the possible paths between two maintenance endpoints should
> > be retrieved, which may quickly become intractable if we apply a naive
> > approach.
> >
> >
> >
> >
> > Thus, multipath routing can be considered to make the network fault-
> > tolerant.  Even better, we can exploit the broadcast nature of
> > wireless networks to exploit meshed multipath routing: we may have
> > multiple Maintenance Intermediate Endpoints (MIP) for each hop in the
> > path.  In that way, each Maintenance Intermediate Endpoint has several
> > possible next hops in the forwarding plane.  Thus, all the possible
> > paths between two maintenance endpoints should be retrieved, which may
> > quickly become untractable if we apply a naive approach.
> >
> > XV>> Same here, IMHO this can be generalized beyond mesh networks. In
> > XV>> 1-hop networks we can still apply OAM methods and exploit
> > XV>> physical redundacy (e.g, dual radio, multiple APs or base
> > XV>> stations, etc..)
> >
> > FT> I use now the term multi-attachment, but I agree; that’s the same
> > concept.  Thank you for raising this distinction.
> >
> >
> >
> >
> > 3.5.  Fault Verification/detection
> >
> > Wired networks tend to present stable performances.  On the contrary,
> > wireless networks are time-variant.  We must consequently make a
> > distinction between _normal_ evolutions and malfunction.
> >
> >
> >
> >
> >
> >
> >
> > Theoleyre, et al.       Expires November 25, 2021               [Page 9]
> >
> > Internet-Draft            OAM features for RAW                  May 2021
> >
> >
> > 3.6.  Fault Isolation/identification
> >
> > The network has isolated and identified the cause of the fault.
> > While DetNet already expects to identify malfunctions, some problems
> > are specific to wireless networks.  We must consequently collect
> > metrics and implement algorithms tailored for wireless networking.
> >
> > For instance, the decrease in the link quality may be caused by
> > several
> > factors: external interference, obstacles, multipath fading, mobility.
> > It it fundamental to be able to discriminate the different causes to
> > make the right decision.
> >
> > 4.  Administration
> >
> > The RAW network has to expose a collection of metrics to support an
> > operator making proper decisions, including:
> >
> > o  Packet losses: the time-window average and maximum values of the
> > number of packet losses have to be measured.  Many critical
> > applications stop to work if a few consecutive packets are dropped;
> >
> > o  Received Signal Strength Indicator (RSSI) is a very common metric
> > in wireless to denote the link quality.  The radio chipset is in
> > charge of translating a received signal strength into a normalized
> > quality indicator;
> >
> > o  Delay: the time elapsed between a packet generation / enqueuing and
> > its reception by the next hop;
> >
> > o  Buffer occupancy: the number of packets present in the buffer, for
> > each of the existing flows.
> >
> > o  Battery lifetime: the expected remaining battery lifetime of the
> > device.  Since many RAW devices might be battery powered, this is an
> > important metric for an operator to take proper decisions.
> >
> > o  Mobility: if a device is known to be mobile, this might be
> > considered by an operator to take proper decisions.
> >
> > These metrics should be collected per device, virtual circuit, and
> > path, as detnet already does.  However, we have to face in RAW to a
> > finer
> > granularity:
> >
> > XV>> I would add also per physical interface. We may have more than
> > XV>> one
> > physical radio technology operating on different bands, etc.
> >
> > FT> you’re entirely right! I updated this part.
> >
> >
> >
> > o  per radio channel to measure, e.g., the level of external
> > interference, and to be able to apply counter-measures (e.g.,
> > blacklisting).
> >
> >
> >
> >
> > Theoleyre, et al.       Expires November 25, 2021              [Page 10]
> >
> > Internet-Draft            OAM features for RAW                  May 2021
> >
> >
> > o  per link to detect misbehaving link (assymetrical link, fluctuating
> > quality).
> >
> > o  per resource block: a collision in the schedule is particularly
> > challenging to identify in radio networks with spectrum reuse.  In
> > particular, a collision may not be systematic (depending on the radio
> > characteristics and the traffic profile)
> >
> > 4.1.  Worst-case metrics
> >
> > RAW inherits the same requirements as DetNet: we need to know the
> > distribution of a collection of metrics.  However, wireless networks
> > are known to be highly variable.  Changes may be frequent, and may
> > exhibit a periodical pattern.  Collecting and analyzing this amount of
> > measurements is challenging.
> >
> > Wireless networks are known to be lossy, and RAW has to implement
> > strategies to improve reliability on top of unreliable links.
> > XV>> to me the following sentence needs more context. It is hard to
> > XV>> understand why you introduce here Hybrid ARQ
> > Hybrid
> > Automatic Repeat reQuest (ARQ) has typically to enable retransmissions
> > based on the end-to-end reliability and latency requirements.
> > FT> Thank you, I reformulated this part ton give more detailed
> > explanations: "Wireless networks are known to be lossy, and RAW has to
> > implement strategies to improve reliability  on top of unreliable links.
> > Reliability is typically achieved through Automatic Repeat Request
> > (ARQ), and Forward Error Correction (FEC).
> > Since the different flows have not the same SLO, RAW must adjust the
> > ARQ and FEC based on the link and path characteristics. "
> >
> >
> >
> > 4.2.  Efficient data retrieval
> >
> > We have to minimize the number of statistics / measurements to
> > exchange:
> >
> > o  energy efficiency: low-power devices have to limit the volume of
> > monitoring information since every bit consumes energy.
> >
> > o  bandwidth: wireless networks exhibit a bandwidth significantly
> > lower than wired, best-effort networks.
> >
> > o  per-packet cost: it is often more expensive to send several packets
> > instead of combining them in a single link-layer frame.
> >
> >
> > In conclusion, we have to take care of power and bandwidth consumption.
> > The following techniques aim to reduce the cost of such
> > maintenance:
> >
> > on-path collection: some control information is inserted in the data
> > packets if they do not fragment the packet (i.e., the MTU is not
> > exceeded).  Information Elements represent a standardized way to
> > handle such information;
> >
> > XV>> Should we consider the IP hop by hop extension headers?
> > FT> good idea: "IP hop by hop extension headers may help to collect
> > metrics all along the path;"
> >
> >
> >
> > flags/fields: we have to set-up flags in the packets to monitor to be
> > able to monitor the forwarding process accurately.  A sequence number
> > field may help to detect packet losses.  Similarly, path
> >
> >
> >
> > Theoleyre, et al.       Expires November 25, 2021              [Page 11]
> >
> > Internet-Draft            OAM features for RAW                  May 2021
> >
> >
> > inference tools such as [ipath] insert additional information in the
> > headers to identify the path followed by a packet a posteriori.
> >
> > hierarchical monitoring; localized and centralized mechanisms have to
> > be combined together.  Typically, a local mechanism should contiuously
> > monitor a set of metrics and trigger distant OAM exchances only when a
> > fault is detected (but possibly not identified).  For instance, local
> > temporary defects must not trigger expensive OAM transmissions.
> >
> > XV>> This is very relevant. The end-to-end reliability, etc.. can also
> > XV>> be
> > a representation of the wireless link performance in an end-to-end
> > path with wired and last mile wireles hop. This is because usually the
> > weaker hop is the wireless one. So having  different levels of
> > monitoring can be a way to go.
> >
> > FT> yes, I agree. I updated the draft to integrate this remark
> > FT> "Besides,
> > the wireless segments represent often the weakest parts of a path: the
> > volume of control information they  produce has to be fixed accordingly.
> > "
> >
> >
> >
> > 4.3.  Reporting OAM packets to the source
> >
> > TODO: statistics are collected when a packet goes from the source to
> > the destination.  However, it has to be also reported by the source.
> > Problem: resource may not be reserved bidirectionnaly.  Even worse:
> > the inverse path may not exist.
> >
> > XV>> To me, having this heirarchical monitoring can be helpful.
> > XV>> Devices
> > may take local decisions when possible and receive end-to-end
> > information when possible.
> >
> > FT>> thank you: "Reporting everything exhaustively to the source may
> > FT>> in
> > most cases too exensive.
> > Thus, devices may take local decisions when possible, and receive
> > end-to- end information when possible.”
> >
> >
> >
> > 5.  Maintenance
> >
> > Maintenance needs to facilitate the maintenance (repairs and upgrades).
> > In wireless networks, repairs are expected to occur much more
> > frequently, since the link quality may be highly time-variant.
> > Thus, maintenance represents a key feature for RAW.
> >
> > 5.1.  Soft transition after reconfiguration
> >
> > Because of the wireless medium, the link quality may fluctuate, and
> > the network needs to reconfigure itself continuously.  During this
> > transient state, flows may begin to be gradually re-forwarded,
> > consuming resources in different parts of the network.  OAM has to
> > make a distinction between a metric that changed because of a legal
> > network change (e.g., flow
> > redirection) and an unexpected event (e.g., a fault).
> >
> > 5.2.  Predictive maintenance
> >
> > RAW needs to implement self-optimization features.  While the network
> > is configured to be fault-tolerant, a reconfiguration may be required
> > to keep on respecting long-term objectives. Obviously, the network
> > keeps on respecting the SLO after a node's crash, but a
> > reconfiguration is required to handle the future faults. In other
> > words, the reconfiguration delay MUST be strictly smaller than the inter-
> fault time.
> >
> > The network must continuously retrieve the state of the network, to
> > judge about the relevance of a reconfiguration, quantifying:
> >
> >
> >
> >
> > Theoleyre, et al.       Expires November 25, 2021              [Page 12]
> >
> > Internet-Draft            OAM features for RAW                  May 2021
> >
> >
> > the cost of the sub-optimality: resources may not be used optimally
> > (e.g., a better path exists);
> >
> > the reconfiguration cost: the controller needs to trigger some
> > reconfigurations.  For this transient period, resources may be twice
> > reserved, and control packets have to be transmitted.
> >
> > Thus, reconfiguration may only be triggered if the gain is significant.
> >
> >
> > Many thanks for all your very constructive comments Xavi!
> >
> > Cheers,
> > Fabrice
> >
> >
> > --
> > RAW mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/raw
> > --
> > RAW mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/raw
> --
> RAW mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/raw