Re: [spring] A belated comment on end-to-end path protection in draft-ietf-spring-resiliency-use-cases

Jeff Tantsura <jefftant.ietf@gmail.com> Tue, 16 May 2017 17:02 UTC

Return-Path: <jefftant.ietf@gmail.com>
X-Original-To: spring@ietfa.amsl.com
Delivered-To: spring@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1C28A12EBE7; Tue, 16 May 2017 10:02:33 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.687
X-Spam-Level:
X-Spam-Status: No, score=-2.687 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jVuHk5JlPOlp; Tue, 16 May 2017 10:02:29 -0700 (PDT)
Received: from mail-io0-x22e.google.com (mail-io0-x22e.google.com [IPv6:2607:f8b0:4001:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C89E312EBF8; Tue, 16 May 2017 09:57:49 -0700 (PDT)
Received: by mail-io0-x22e.google.com with SMTP id o12so97756688iod.3; Tue, 16 May 2017 09:57:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=user-agent:date:subject:from:to:cc:message-id:thread-topic :references:in-reply-to:mime-version; bh=NEPc8p4aBvwlmdrKjG8Qq2bM6JQfdaylJJjlPeSJobc=; b=LW6OdGPvIjLFZwUiOdXy2oHm1uegYJjpDKXsFVAaBVPwQ/UogixaqaFB962oXAablY BRUaBcYQVHFlDudbKk+EJq14sWtQyEo9GlteJbtTwT8bAoBN78l2Lz46j2BFvHwYhgTV XwUgtDtDKj3W9uaoFSEAZ1K0l7rHUr5slT2/U9yv4Jyi4ytepKagRHsY99Ifau0/ILZl 7En0FDwuRBJdt+Rd8PKuvxPTNNYmfp2KVS7zoko+Ofx5UrN0jq35sxSBkW49x5pb7fm0 RiEz0rGjEGOXxxpO4iD1ZOAUmE40OSJGpKUR1U+oJNPPFQ/p+Bsu6OaB3kOje+WZH9Wu xixA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:user-agent:date:subject:from:to:cc:message-id :thread-topic:references:in-reply-to:mime-version; bh=NEPc8p4aBvwlmdrKjG8Qq2bM6JQfdaylJJjlPeSJobc=; b=C6Orzj935ymcFGWFbezYtDBp9SnGGnPnglR20ur51DZ/HdbY2WuhlrlUD5vG1aH6dj t8ww70APdbeY8xZIrv+H3o+b7nIF1nrfuSpNxQQ9lkCZDPbvnruQWbZ9oV1+YYTC1DIv 1BsDUnOxwkTgcjVqG8Mcb769o2AqyWJ43FYc7BRgA7s85cyWAfC8ygLG0sqE8zFLTT1c 9ijpAg0A/OwJkDG+YykRkp8sE9nSZHOrI+kBURDB+uHMkEgxSgIiH5xs6jTbWv33Whdg 8wgg0Pf6gPK9itnoh17Grni4ixoiegCwofbUheBVuzax8bhyQzRCr4jZ/yhMBLR9SWqa i50g==
X-Gm-Message-State: AODbwcCrMnct3U/HYSHu7lnHkwHTBbzEG0MGcJJsYSv61+iF3gBGf8g0 xUU+jywKeL5GOpLF
X-Received: by 10.107.151.72 with SMTP id z69mr11884120iod.98.1494953868964; Tue, 16 May 2017 09:57:48 -0700 (PDT)
Received: from [172.20.9.56] (marriott-chateau-champlain-montreal.sites.intello.com. [66.171.169.34]) by smtp.gmail.com with ESMTPSA id n22sm1184651itg.25.2017.05.16.09.57.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 May 2017 09:57:47 -0700 (PDT)
User-Agent: Microsoft-MacOutlook/f.21.0.170409
Date: Tue, 16 May 2017 09:57:45 -0700
From: Jeff Tantsura <jefftant.ietf@gmail.com>
To: Alexander Vainshtein <Alexander.Vainshtein@ecitele.com>
CC: "spring@ietf.org" <spring@ietf.org>, "draft-ietf-spring-resiliency-use-cases@ietf.org" <draft-ietf-spring-resiliency-use-cases@ietf.org>
Message-ID: <FD404255-6901-4A6B-BB90-62D3A9F45DEB@gmail.com>
Thread-Topic: [spring] A belated comment on end-to-end path protection in draft-ietf-spring-resiliency-use-cases
References: <AM4PR03MB1713393C262301279EAF29039DED0@AM4PR03MB1713.eurprd03.prod.outlook.com> <4CE8B71E-1CB7-43AF-9DA3-D936E030A2CA@cisco.com> <AM4PR03MB1713F46B5662731126099CFE9DE60@AM4PR03MB1713.eurprd03.prod.outlook.com> <CAKz0y8wPO6VcMJ6Ba_m1A5L2F5bh2rv7761C8vGo51H+xSRfuA@mail.gmail.com> <AM4PR03MB1713AAD69441A6C92D63B5919DE60@AM4PR03MB1713.eurprd03.prod.outlook.com> <CAKz0y8zHzneU4SUtH8RGp0kjKVc=XfFZ3uO6e8NNFGn0X383LQ@mail.gmail.com> <AM4PR03MB171393C194C01D56F00513E59DE60@AM4PR03MB1713.eurprd03.prod.outlook.com> <CAKz0y8xnkL10YFr7+V8i5ECe0Zgzr7hELgKnHjDxm5WgOzdPjQ@mail.gmail.com> <AM4PR03MB17134890B40531136D7EE5719DE60@AM4PR03MB1713.eurprd03.prod.outlook.com> <15C96387-0796-4E0A-9C79-E5B6576D2829@gmail.com>
In-Reply-To: <15C96387-0796-4E0A-9C79-E5B6576D2829@gmail.com>
Mime-version: 1.0
Content-type: multipart/alternative; boundary="B_3577773466_1291860587"
Archived-At: <https://mailarchive.ietf.org/arch/msg/spring/KKp9eMM7YTLlxznN03iWBJ-pzQ4>
Subject: Re: [spring] A belated comment on end-to-end path protection in draft-ietf-spring-resiliency-use-cases
X-BeenThere: spring@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Stacked Tunnels for Source Routing \(STATUS\)." <spring.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/spring>, <mailto:spring-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/spring/>
List-Post: <mailto:spring@ietf.org>
List-Help: <mailto:spring-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/spring>, <mailto:spring-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 16 May 2017 17:02:33 -0000

resending with reduced number of recipients.

 

Cheers,

Jeff

 

Sasha,

 

Don’t forget – RSVP-TE FRR has explicit signaling and state associated with it, as well as well defined state transitions, SR on contrary doesn’t. 

Changes in topology (link/node down events) are not communicated back to the head-end directly but rather flooded thru a routing protocol (for sake of this discussion lets ignore the possibility of running fast failure detection over SR tunnels).

Trying to derive operation state at PLR based on transitional changes in a routing protocol is a rather complicated task.

 

Cheers,

Jeff

 

 

From: spring <spring-bounces@ietf.org> on behalf of Alexander Vainshtein <Alexander.Vainshtein@ecitele.com>
Date: Tuesday, May 16, 2017 at 09:23
To: Muthu Arul Mozhi Perumal <muthu.arul@gmail.com>
Cc: "spring@ietf.org" <spring@ietf.org>, Shell Nakash <Shell.Nakash@ecitele.com>, Michael Gorokhovsky <Michael.Gorokhovsky@ecitele.com>, "draft-ietf-spring-resiliency-use-cases@ietf.org" <draft-ietf-spring-resiliency-use-cases@ietf.org>, Sidd Aanand <Sidd.Aanand@ecitele.com>, "Stefano Previdi (sprevidi)" <sprevidi@cisco.com>, Ron Sdayoor <Ron.Sdayoor@ecitele.com>, Rotem Cohen <Rotem.Cohen@ecitele.com>
Subject: Re: [spring] A belated comment on end-to-end path protection in draft-ietf-spring-resiliency-use-cases

 

Muthu,

Again lots of thanks for a prompt response.

 

We seem to agree on the following points:

·         In SR some failures cannot be handled by local protection (actually, there is an expired draft that defines how this could be done, but it introduces serious complexity)

·         Combining local protection with end-to-end path protection is possible. In particular, such a combination speeds up handling of failures that that can be handled locally while also handling failures that could not be addressed by local protection.

 

Whether combining both forms of protection carries with it some new problems or not is a different story.

 

Regards,

Sasha

 

Office: +972-39266302

Cell:      +972-549266302

Email:   Alexander.Vainshtein@ecitele.com

 

From: Muthu Arul Mozhi Perumal [mailto:muthu.arul@gmail.com] 
Sent: Tuesday, May 16, 2017 7:11 PM
To: Alexander Vainshtein <Alexander.Vainshtein@ecitele.com>
Cc: Stefano Previdi (sprevidi) <sprevidi@cisco.com>; spring@ietf.org; Shell Nakash <Shell.Nakash@ecitele.com>; Michael Gorokhovsky <Michael.Gorokhovsky@ecitele.com>; draft-ietf-spring-resiliency-use-cases@ietf.org; Sidd Aanand <Sidd.Aanand@ecitele.com>; Ron Sdayoor <Ron.Sdayoor@ecitele.com>; Rotem Cohen <Rotem.Cohen@ecitele.com>
Subject: Re: [spring] A belated comment on end-to-end path protection in draft-ietf-spring-resiliency-use-cases

 

Sasha,

 

On Tue, May 16, 2017 at 4:29 PM, Alexander Vainshtein <Alexander.Vainshtein@ecitele.com> wrote:

Muthu,

An additional clarification:

·         If the link BC were OK, B could pop B from the stack and send packets to C with just D in the stack

·         When the link BC fails, B will leave the stack as (CD) IMHO – it would be  just trying to bypass the failed link BC.

·         If the failure of BC as perceived by B was cause by the failure of node B, such a failure could not be recovered by local protection. This is exactly the scenario where local protection for shortest SR path comprising an SR-TE path should be augmented by end-to-end path protection.

​If node B fails, the e2e path monitoring at  node A would anyway detect the failure and switch the traffic over an alternate disjoint path...​

 

Regarding combination of local protection with end-to-end protection for RSVP-TE – AFAIK this was never used because it would not provide any added value. 

In SR this is not so because local protection is usually faster (and scales better) than end-to-end protection, but, as opposed to RSVP-TE, there are failures that local protection cannot fix.

 

​Agree, there are failures in SR-TE that local protection cannot fix as desired, so it calls for e2e path protection. However, enabling them together is not always the best approach since it can introduce other problems to solve.

 

Regards,

Muthu

 

 

Regards,

Sasha

 

Office: +972-39266302

Cell:      +972-549266302

Email:   Alexander.Vainshtein@ecitele.com

 

From: Alexander Vainshtein 
Sent: Tuesday, May 16, 2017 1:42 PM
To: 'Muthu Arul Mozhi Perumal' <muthu.arul@gmail.com>
Cc: Stefano Previdi (sprevidi) <sprevidi@cisco.com>; spring@ietf.org; Shell Nakash <Shell.Nakash@ecitele.com>; Michael Gorokhovsky <Michael.Gorokhovsky@ecitele.com>; draft-ietf-spring-resiliency-use-cases@ietf.org; Sidd Aanand <Sidd.Aanand@ecitele.com>; Ron Sdayoor <Ron.Sdayoor@ecitele.com>; Rotem Cohen <Rotem.Cohen@ecitele.com>
Subject: RE: [spring] A belated comment on end-to-end path protection in draft-ietf-spring-resiliency-use-cases

 

Muthu,

Again lots of thanks for a prompt response. I still do not think a loop would really form because:

·         A sends packet to its local next hop for B with the stack (B, C, D)

·         B receives this packet with the stack (C, D), but the link C has failed. So B sends to its next hop for it back to A with stack (C,D)

·         A now sends the packet to its next hop for C with the same stack.

 

Regards,

Sasha

 

Office: +972-39266302

Cell:      +972-549266302

Email:   Alexander.Vainshtein@ecitele.com

 

From: Muthu Arul Mozhi Perumal [mailto:muthu.arul@gmail.com] 
Sent: Tuesday, May 16, 2017 1:25 PM


To: Alexander Vainshtein <Alexander.Vainshtein@ecitele.com>
Cc: Stefano Previdi (sprevidi) <sprevidi@cisco.com>; spring@ietf.org; Shell Nakash <Shell.Nakash@ecitele.com>; Michael Gorokhovsky <Michael.Gorokhovsky@ecitele.com>; draft-ietf-spring-resiliency-use-cases@ietf.org; Sidd Aanand <Sidd.Aanand@ecitele.com>; Ron Sdayoor <Ron.Sdayoor@ecitele.com>; Rotem Cohen <Rotem.Cohen@ecitele.com>
Subject: Re: [spring] A belated comment on end-to-end path protection in draft-ietf-spring-resiliency-use-cases

 

On Tue, May 16, 2017 at 3:27 PM, Alexander Vainshtein <Alexander.Vainshtein@ecitele.com> wrote:

Muthu,

Lots of thanks for a prompt response.

 

I do not think that the loop you have described would actually appear in the scenario you’ve described.

 

To the best of my understanding of TI-LFA, B would send the traffic back to A complete with an explicit route that says Bà Aà CàD, and no loop would be formed.  

 

Not necessarily. B was asked to send the traffic to C and knows that if it sends the traffic to A, then A will send it to C over the shortest path (i.e from B's perspective only the labeled next-hop changes). Unfortunately, A has an explicit route pointing back to B (over the SR-TE tunnel T1) that B isn't aware of. If B does strict explicit route for everything, then B can run out of its MSD.. 

 

​

 

Similar “loops” can happen also in MPLS FRR with RSVP-TE when the PLR sends some traffic back  - but it sends it with the suitable label stack of the bypass tunnel so that eventually it reaches the MP.

 

​Are there existing deployments where both e2e path protection and local protection are used together with RSVP-TE?

 

Regards,

Muthu

 

 

Regards,

Sasha

 

Office: +972-39266302

Cell:      +972-549266302

Email:   Alexander.Vainshtein@ecitele.com

 

From: Muthu Arul Mozhi Perumal [mailto:muthu.arul@gmail.com] 
Sent: Tuesday, May 16, 2017 12:34 PM
To: Alexander Vainshtein <Alexander.Vainshtein@ecitele.com>
Cc: Stefano Previdi (sprevidi) <sprevidi@cisco.com>; spring@ietf.org; Shell Nakash <Shell.Nakash@ecitele.com>; Michael Gorokhovsky <Michael.Gorokhovsky@ecitele.com>; draft-ietf-spring-resiliency-use-cases@ietf.org; Sidd Aanand <Sidd.Aanand@ecitele.com>; Ron Sdayoor <Ron.Sdayoor@ecitele.com>; Rotem Cohen <Rotem.Cohen@ecitele.com>


Subject: Re: [spring] A belated comment on end-to-end path protection in draft-ietf-spring-resiliency-use-cases

 

Using end-to-end path protection together with local protection can result in traffic loops. Consider the foll. topology:

 

B-----C   

|    / \

|   /   \ 

|  /     \

| /       \D----+ 

A/              Z (CE)

 \         F----+ 

  \       /

   \     /

    \   / 

     \E/

 

- All links are of equal cost.

- A, D and F are BGP peers.

- Z is a dual-homed CE.

 

A resolves its BGP next-hop D over the SR-TE tunnel T1.

T1: A->B, B->C, C->D (loosely routed)

 

Suppose A has enabled end-to-end path protection over tunnel T1 and B has TI-LFA enabled, and the detection timers are configured as described in your previous email. If the BC link goes down, B will immediately start rerouting the traffic via A (in FRR fashion) creating a loop b/w A and B.

 

A solution would be to make the A-B link ineligible for TI-LFA backup computation at B. However, managing this network-wide could become operational expensive. Hence, deploying one of end-to-end path protection or local protection with sufficiently short detection timers keeps things simple, IMHO.

 

Regards,

Muthu

 

On Tue, May 16, 2017 at 1:59 PM, Alexander Vainshtein <Alexander.Vainshtein@ecitele.com> wrote:

 

 

Regards,

Sasha

 

Office: +972-39266302

Cell:      +972-549266302

Email:   Alexander.Vainshtein@ecitele.com

 

From: Alexander Vainshtein 
Sent: Tuesday, May 16, 2017 11:28 AM
To: 'Stefano Previdi (sprevidi)' <sprevidi@cisco.com>
Cc: draft-ietf-spring-resliency-use-cases@ietf.org; spring@ietf.org; Shell Nakash <Shell.Nakash@ecitele.com>; Michael Gorokhovsky <Michael.Gorokhovsky@ecitele.com>; Sidd Aanand <Sidd.Aanand@ecitele.com>; Ron Sdayoor <Ron.Sdayoor@ecitele.com>; Rotem Cohen <Rotem.Cohen@ecitele.com>
Subject: RE: [spring] A belated comment on end-to-end path protection in draft-ietf-spring-resiliency-use-cases

 

Stefano,

Lots of thanks for a prompt response.

 

A couple of short comments if you do not mind:

 

Using 2119 language in a "use cases" document: 

1.       Going back to the source I see that “MUST NOT… mean that the definition is an absolute prohibition of the specification”

2.       I agree that the use case document defines which scenarios should be addressed, but I do not see how it can impose an absolute prohibition on a certain scenario. 

 

Little sense link protection has in the case of path protection:

1.       This was definitely correct for traditional traffic engineering because the “shortest traffic paths” (e.g., LDL PSPs) could be easily differentiated from the “engineered traffic paths”. 

2.       In addition, traditional local protection (e.g., MPLS FRR using RSVP-TE) could deal with link and node failures regardless of whether the failed link or node appeared in the ERO of the protected path.

3.       IMHO and FWIW, with SR  the situation is quite different:

o   The shortest traffic paths not only coexist with engineered traffic paths: the latter are in many cases “tunneled” within the former.

o   Path protection cannot be applied to shortest traffic paths so they must rely on local protection

o   Local protection in the case of failure of a node or link that appears in the ERO of an engineered SR path is highly non-trivial at best, so path protection for the engineered LSPs looks like a preferred solution to me.

I fully agree with you that the operators deploying SR should provide feedback on this point based on actual operational experience. 

Meanwhile I doubt that a priori declaring some use cases as absolutely prohibited is the right thing to do.

 

My 2c,

Sasha

 

Office: +972-39266302

Cell:      +972-549266302

Email:   Alexander.Vainshtein@ecitele.com

 

 

-----Original Message-----
From: Stefano Previdi (sprevidi) [mailto:sprevidi@cisco.com] 
Sent: Monday, May 15, 2017 11:12 AM
To: Alexander Vainshtein <Alexander.Vainshtein@ecitele.com>
Cc: draft-ietf-spring-resliency-use-cases@ietf.org; spring@ietf.org; Shell Nakash <Shell.Nakash@ecitele.com>; Michael Gorokhovsky <Michael.Gorokhovsky@ecitele.com>; Sidd Aanand <Sidd.Aanand@ecitele.com>; Ron Sdayoor <Ron.Sdayoor@ecitele.com>; Rotem Cohen <Rotem.Cohen@ecitele.com>
Subject: Re: [spring] A belated comment on end-to-end path protection in draft-ietf-spring-resiliency-use-cases

 

 

> On May 11, 2017, at 12:04 PM, Alexander Vainshtein <Alexander.Vainshtein@ecitele.com> wrote:

> 

> Hi all,

> I have a belated (but hopefully late is still better than never) comment on path protection as defined in Section 2 of the draft.

>  

> This second para in this section says:

>    A first protection strategy consists in excluding any local repair

> 

>    but instead use end-to-end path protection where each SPRING path 

> is

> 

>    protected by a second disjoint SPRING path.  In this case local

> 

>    protection MUST NOT be used.

> 

> First of all, I do not think that RFC 2119 language should be used in Informational documents, especially in the documents that describe use cases.

 

 

this document is also a requirements document for the resiliency use-case. RFC2119 terminology is perfectly usable and even more, it adds clarity on what the solution is expected to provide.

 

 

> In addition, I specifically disagree with the quoted statement above, because, from my POV:

> ·         Local repair and end-to-end path protection can be combined for the same path

> ·         Such a combination may be beneficial for the operators.

 

 

are you talking by experience or is it just something that came into your mind ? I’d like to hear from operators using a combination of path and link protection.

 

This document has been deeply reviewed also by operators and it has been always obvious the little sense link protection has in case of path protection.

 

 

> One possible way to combine the two is described below:

>  

> 1.       A pair of SR paths is set up between the given two nodes – later referred to as source and destination -  in the network. These paths are “SR-disjoint” in the sense that their “explicit routes”  do not have any common elements, be they nodes or adjacencies, with exclusion of the final destination

> 2.       Local repair for these paths is enabled in the network. It is triggered by locally observed events (link failures etc.), applied by the nodes adjacent to the failure and guarantees that, in the case of a link or node failure that is not specified in the explicit route, traffic along the affected path would be restored within <X> milliseconds

> 3.       End-to-end liveness monitoring is enabled for the two SR paths, and detects end-to-end failures of these paths within <Y> milliseconds where Y >> X. In other words, end-to-end liveness monitoring for these paths will ignore any failures that local repair can fix, but will detect failures that cannot be locally repaired (e.g., failures of nodes or links that have been specified in the explicit route of one of the paths

> 4.       End-to-end liveness monitoring triggers end-to-end path protection to be applied by the source node in the following way:

> a.       If it recognizes both paths as alive, one of them will carry the customer traffic, while the other one will be idle. The rules for selecting the active path in this scenario may vary

> b.      If end-to-end failure of one of these paths is detected while the other one remains alive, traffic will be carried across the live path

> c.       If end-to-end failure of both paths is detected (e.g., if the final destination node fails, or if the network is partitioned), this is recognized as an unrecoverable failure.

>  

> From my POV the combination of local repair and end-to-end protection for SR paths is one of a few possibilities to protect such paths against failures of nodes and/or links that have been specified in their explicit routes. (Another option has been described in Node Protection for SR-TE Paths, but this draft has expired).

>  

> Do I miss something substantial?

 

 

to my view you created a use-case that doesn’t bring much to the picture but I’d let operators to comment.

 

s.

 

 

>  

> Regards,

> Sasha

>  

> Office: +972-39266302

> Cell:      +972-549266302

> Email:   Alexander.Vainshtein@ecitele.com

>  

> 

> ______________________________________________________________________

> _____

> 

> This e-mail message is intended for the recipient only and contains 

> information which is CONFIDENTIAL and which may be proprietary to ECI 

> Telecom. If you have received this transmission in error, please 

> inform us by e-mail, phone or fax, and then delete the original and all copies thereof.

> ______________________________________________________________________

> _____ _______________________________________________

> spring mailing list

> spring@ietf.org

> https://www.ietf.org/mailman/listinfo/spring

 


___________________________________________________________________________

This e-mail message is intended for the recipient only and contains information which is 
CONFIDENTIAL and which may be proprietary to ECI Telecom. If you have received this 
transmission in error, please inform us by e-mail, phone or fax, and then delete the original 
and all copies thereof.
___________________________________________________________________________


_______________________________________________
spring mailing list
spring@ietf.org
https://www.ietf.org/mailman/listinfo/spring

 


___________________________________________________________________________

This e-mail message is intended for the recipient only and contains information which is 
CONFIDENTIAL and which may be proprietary to ECI Telecom. If you have received this 
transmission in error, please inform us by e-mail, phone or fax, and then delete the original 
and all copies thereof.
___________________________________________________________________________

 


___________________________________________________________________________

This e-mail message is intended for the recipient only and contains information which is 
CONFIDENTIAL and which may be proprietary to ECI Telecom. If you have received this 
transmission in error, please inform us by e-mail, phone or fax, and then delete the original 
and all copies thereof.
___________________________________________________________________________

 


___________________________________________________________________________

This e-mail message is intended for the recipient only and contains information which is 
CONFIDENTIAL and which may be proprietary to ECI Telecom. If you have received this 
transmission in error, please inform us by e-mail, phone or fax, and then delete the original 
and all copies thereof.
___________________________________________________________________________
_______________________________________________ spring mailing list spring@ietf.org https://www.ietf.org/mailman/listinfo/spring