From nobody Wed Mar 10 21:49:16 2021
Return-Path: <hayabusagsm@gmail.com>
X-Original-To: lsr@ietfa.amsl.com
Delivered-To: lsr@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1])
 by ietfa.amsl.com (Postfix) with ESMTP id DB9B53A110C
 for <lsr@ietfa.amsl.com>; Wed, 10 Mar 2021 21:49:13 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.997
X-Spam-Level: 
X-Spam-Status: No, score=-1.997 tagged_above=-999 required=5
 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
 DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=0.1,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001,
 URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key)
 header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44])
 by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id wg0iJ-uA4OCP for <lsr@ietfa.amsl.com>;
 Wed, 10 Mar 2021 21:49:10 -0800 (PST)
Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com
 [IPv6:2607:f8b0:4864:20::535])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (No client certificate requested)
 by ietfa.amsl.com (Postfix) with ESMTPS id 6AE563A110A
 for <lsr@ietf.org>; Wed, 10 Mar 2021 21:49:10 -0800 (PST)
Received: by mail-pg1-x535.google.com with SMTP id p21so12971435pgl.12
 for <lsr@ietf.org>; Wed, 10 Mar 2021 21:49:10 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; 
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=7AOKCi3ox5o8KngiGQI+nCpgVqF7lDETRtFWhY+VMqk=;
 b=FIlP4NdrBcd97TcbChCGVyGsjZTBk3ylUm8iyr9szyooiM8MkGcBbUn+PY1i6CRHSb
 gtInF0fvjwBySPBGGOpPw/kKw1tr6/dO8Nb7zDiFMA5G8Xsl+scvmmtQjJvWWnhSwkhs
 Ea6rqwn/XkmyTuB8S7QEA8PTgFZQzJqPkjy2nVJFtJk91PTnZo8NwEme7RSt18gOVcPn
 7DzppAZCooTHg0EJRStbxjOD5UZpOBtr2CyRvjYSj72prOP6w8Wcp+KUaXpxWNEEm2DL
 53B4WTNqpKUnniWbh1s7RKT7cMaAbTk/PGgHlkWzDJDVnTRWGOGGodiFpuuCawoSWSMP
 JVkg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=7AOKCi3ox5o8KngiGQI+nCpgVqF7lDETRtFWhY+VMqk=;
 b=aC4WhoCEUMXnG9TSfhIHc6XQm23CqJAPhDY9SQGhQ6thGG90c18w7Bi2LPvjhB0ze5
 XDvABf2H9PN9y9hJxZkzW1smDZezdX8a534fWznd3aCu8FTY/cXi/HR9CBpCO2i70fRS
 wk0E0ysZiXKC7OK0XnWiO5p8/V2sQkB2JQQOUgy3gCBl5ub+E4H7oDCGygVY+6yLlnNh
 abc8boa/aNkPjd13eeEUtjXTZTCMeaWxGm6vip0bWM3GiWK3t7m5e1eKFnOtmNqN9jTN
 gSzP0DbCuGL76f9fFZXZLbUFQWIjyKlB383QWcctIMpoXwRfDhhCESX/HVtyPlTNwhit
 R7CA==
X-Gm-Message-State: AOAM532Cm9fPgvuS0e9RuZo7tX4ymeLOdrdNmvnP+ok6PHURiORnJhu8
 ExG2kzHRcNL104w+cWMbQAHjbbaCbep6us3389A=
X-Google-Smtp-Source: ABdhPJyCWKFwLhdmRe6koHWGAF6eTodNIT50SXZyueWjqIi4XQZnLcMH4tp6Pb3T3l4XTbXGHMvnpu4NfvyeCdBSFP4=
X-Received: by 2002:a05:6a00:168d:b029:1ba:d500:1209 with SMTP id
 k13-20020a056a00168db02901bad5001209mr6374465pfc.4.1615441748507; Wed, 10 Mar
 2021 21:49:08 -0800 (PST)
MIME-Version: 1.0
References: <SN6PR13MB23348360D4BE7D6E73B5828385929@SN6PR13MB2334.namprd13.prod.outlook.com>
 <8207CCF7-7C42-429E-B368-CA9CD99DE06D@chopps.org>
 <383410c728b948e194538960987f8468@huawei.com>
 <CABNhwV1MSpMXY+=-jKfFrozd2=ztZ38SybxDM6_LVFRVHGR48g@mail.gmail.com>
 <SN6PR13MB23345877A0012E67EA65CB5885919@SN6PR13MB2334.namprd13.prod.outlook.com>
In-Reply-To: <SN6PR13MB23345877A0012E67EA65CB5885919@SN6PR13MB2334.namprd13.prod.outlook.com>
From: Gyan Mishra <hayabusagsm@gmail.com>
Date: Thu, 11 Mar 2021 00:48:57 -0500
Message-ID: <CABNhwV1Nor-+iHJXPHF09Kx-KB2xJ78Lz7ho3zEjOCWkAch-wA@mail.gmail.com>
To: Linda Dunbar <linda.dunbar@futurewei.com>
Cc: Christian Hopps <chopps@chopps.org>, Liyizhou <liyizhou@huawei.com>, 
 "lsr@ietf.org" <lsr@ietf.org>
Content-Type: multipart/alternative; boundary="00000000000093428c05bd3c57e3"
Archived-At: <https://mailarchive.ietf.org/arch/msg/lsr/OVR5elul8smmc-5Y7wFY4owVWsA>
Subject: Re: [Lsr] Why not leverage Network conditions to optimize balancing
 among multiple App Layer Load Balancers? as proposed by
 draft-dunbar-lsr-5g-edge-compute-ospf-ext
X-BeenThere: lsr@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Link State Routing Working Group <lsr.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/lsr>,
 <mailto:lsr-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/lsr/>
List-Post: <mailto:lsr@ietf.org>
List-Help: <mailto:lsr-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/lsr>,
 <mailto:lsr-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 11 Mar 2021 05:49:14 -0000

--00000000000093428c05bd3c57e3
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Linda

Comments in-line

On Wed, Mar 10, 2021 at 6:46 PM Linda Dunbar <linda.dunbar@futurewei.com>
wrote:

> Gyan,
>
>
>
> To a router, having multiple servers with the same (ANYCAST) address
> attached to different egress routers (A-ER) is same as having multiple
> paths to reach the (ANYCAST) address.
>
>
>
> You are absolutely correct that there are many tools to influence the pat=
h
> section, such as the routing distance, TE metrics, policies, etc.
>
>
>
> draft-dunbar-lsr-5g-edge-compute-ospf-ext proposes to add another
> component to influence the path selection: the =E2=80=9CSite-Cost=E2=80=
=9D  which is
> influenced by  =E2=80=9Csite-capacity + load measurement + Preference + x=
xx=E2=80=9D. The
> =E2=80=9Csite-Cost=E2=80=9D can be raw measurements collected by the egre=
ss routers based
> on the instruction from a controller, or informed by the App Controller
> periodically.
>
>
>
> In the past, ANYCAST has been predominantly used for multiple servers in
> geographically far apart locations so that routing distance alone can
> always nail down to one specific ANYCAST server.
>
>  Gyan>  The Anycast environments that I have worked with architecture hav=
e
> been server clusters in data centers geographically diverse that sit behi=
nd
> a load balancer that uses a concept called HRI host route injection that =
if
> the service is up the VIP is BGP advertised for the cluster to DC core an=
d
> then services VIP host route are advertised into the core all BGP
> attributes equal so lowest IGP metric tie breaker picks best path shortes=
t
> path across the core  as best path and of iBGP multipath is used that
> services VIPs are geographically load balanced flow based if metric is a
> tie and if IPv6 is used in the core then 5-tuple per flow load balanced
> optimized.
>


3GPP TR23.748 (5G Edge Computing) is proposing to use multiple servers (or
> multiple App Layer Load Balancers) with the same ANYCAST address in their
> Local IP Data Network to avoid the single point of failure and the
> bottleneck at the App Layer Load Balancer for mission critical
> applications.
>
>  Gyan>  With Anycast routing as I mentioned you don=E2=80=99t have a sing=
le point
> of failure and really have optimal redundancy which is why for any servic=
es
> such as DNS, NTP and many others Anycast is the best most redundancy and
> optimal as proximity routing is used. If your closest lowest IGP metric
> iBGP load balanced path goes down let=E2=80=99s say multiple DC outages y=
ou still
> have your next closest in the BGP path list pecking order to reach the
> service thus optimized redundancy as every DC would have to be down for
> service VIP to be down.  Also with Anycast as it a distributed architectu=
re
> you are not overloading core paths or certain DC VIPs as all traffic is
> distributed geographically proximity load balanced automatically.  So the=
re
> is a lesser chance of bottleneck as the Anycast architecture is distribut=
ed.
>

   As far as 5G you still have main components of the path from UE  user
data plane to RAN xHaul VPN within the wireless operator network then
handoff to the core to a service VIP in a closet proximity data center. So
now we are trying to further optimize the cost based on real time PM
performance metrics to calculate the best path with this draft.

This draft below is in LSR maybe interesting to you related to flex algo
bandwidth related constraints so that the TE static ERO concept of exclude
L2 links in a bundle can be accomplished based on link delay bandwidth
thresholds that use semi dynamic metrics based on PM measurements.

In that discussion thread was brought up use of the PM based metrics and if
that would cause instability.  That maybe something to consider when using
dynamic PM based metrics.

https://datatracker.ietf.org/doc/html/draft-hegde-lsr-flex-algo-bw-con-01

Kind Regards


Gyan

Thank you very much for your comments. I have made some changes to the
> text. Please see the revision:
> https://datatracker.ietf.org/doc/draft-dunbar-lsr-5g-edge-compute-ospf-ex=
t/
>
>
>
>
>
> Linda
>
>
>
> *From:* Gyan Mishra <hayabusagsm@gmail.com>
> *Sent:* Tuesday, March 9, 2021 12:08 AM
> *To:* Liyizhou <liyizhou@huawei.com>
> *Cc:* Christian Hopps <chopps@chopps.org>; Linda Dunbar <
> linda.dunbar@futurewei.com>; lsr@ietf.org
> *Subject:* Re: [Lsr] Why not leverage Network conditions to optimize
> balancing among multiple App Layer Load Balancers? as proposed by
> draft-dunbar-lsr-5g-edge-compute-ospf-ext
>
>
>
> Linda and authors
>
>
>
> Some thoughts regarding load balancing draft.
>
>
>
> Anycast in my experience has been used predominantly in my experience
> within operators networks with BGP overlay,  using BGP best path selectio=
n
> and most cases boils down to lowest IGP metric tie breaker shortest path
> for the service Anycast proximity route which you can also with unique RD
> in overlay and can take advantage of iBGP multipath equal cost load
> balancing over an operator vaccine or 4G/5G RAN xhaul or internet.
>
>
>
> The nice thing about Anycast with BGP overlay you as are automatically
> proximity based routing load balancing inherent to Anycast routing.
>
>
>
> Point here is we are using BGP best path selection but it does boils down
> to IGP lowest metric tie breaker but you can use iBGP multipath to furthe=
r
> optimize the routing for cloud computing.
>
>
>
> We have so many tools in our operators toolbox to optimize routing SR or
> Flex-Algo, SDN etc am wondering if some form of SDN or SD-WAN overlay cou=
ld
> provide the Dyncast type Dynamic Anycast solution.
>
>
>
> I want to the wiki page for Dyncast.  The presentation is not available
> yet.  Will check tomorrow.
>
>
>
> Thanks
>
>
>
> Gyan
>
>
>
>
>
> On Mon, Mar 8, 2021 at 11:36 PM Liyizhou <liyizhou@huawei.com> wrote:
>
> Hi,
>
>
>
> Sorry to chime in.
>
>
>
> There are certainly some higher layer application/protocols to employ. At
> the same time, there are some advantages of network layer approaches as
> well in my mind.
>
>
>
> When talking about edge computing, there are some unique characteristics.
> The number of edge sites could be large or huge in future in a big city.
> Edges are geographically scattered which could be a few, or tens of, or a
> hundred kilometers away from each other, and each site has limited
> computing resources which could be a small cluster. Application layer bas=
ed
> approach normally would rely on one or several =E2=80=9Cserver=E2=80=9D/=
=E2=80=9Dbroker=E2=80=9D to be
> responsible for request handling all over the city. As such =E2=80=9Cserv=
ers=E2=80=9D are
> unlikely available on each and every edge site, it introduces additional
> path stretch for data packets requiring delivery to other edge sites firs=
t.
> Such path stretch introduces additional (network and processing) delay
> which could be crucial for short live request flow. On the contrary, the
> network node at the edge is naturally sitting on the data path without
> introducing any additional cost to direct the (explicit/implicit) request
> somewhere else. Also routing system has been proven doing good in such
> distributed manner.
>
>
>
>
>
> There is a dyncast (dynamic anycast) work ongoing. It is not exactly same
> as what Linda proposed here, but some relations can be seen, like trying =
to
> use anycast methodology to access an edge computing, especially
> computational intensive, service. The current discussions are about
> compellingness of the use cases, the deficiency of existing solutions, an=
d
> proposed architecture, not gone very far into what specific routing
> protocols to use yet. A side meeting will be held on Wed 10am CET. You ma=
y
> check https://github.com/dyncast/ietf110
> <https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgith=
ub.com%2Fdyncast%2Fietf110&data=3D04%7C01%7Clinda.dunbar%40futurewei.com%7C=
090a578c2711455d431808d8e2c1ad0b%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0=
%7C637508668779221152%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV=
2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3D%2BARHREw51PPjAyjut9I9=
Z5Ym%2FV%2FDYwPl2xO2luZxIag%3D&reserved=3D0>
> for more info.
>
>
>
> Cheers,
>
> Yizhou
>
>
>
> *From:* Lsr [mailto:lsr-bounces@ietf.org] *On Behalf Of *Christian Hopps
> *Sent:* Tuesday, March 9, 2021 9:00 AM
> *To:* Linda Dunbar <linda.dunbar@futurewei.com>
> *Cc:* lsr@ietf.org; Christian Hopps <chopps@chopps.org>
> *Subject:* Re: [Lsr] Why not leverage Network conditions to optimize
> balancing among multiple App Layer Load Balancers? as proposed by
> draft-dunbar-lsr-5g-edge-compute-ospf-ext
>
>
>
>
>
>
>
> On Mar 8, 2021, at 7:40 PM, Linda Dunbar <linda.dunbar@futurewei.com>
> wrote:
>
>
>
> Christian,
>
>
>
> You said at LSR session today that there might be concern of network
> optimizing ANYCAST traffic to better balance among multiple App Layer Loa=
d
> Balancers.
>
> First of all, only the Applications that need to leverage the network
> condition to balance among their multiple Load Balancers will get the
> benefit of path selection that are based on the combination of routing
> distance and other dynamic running status. The networks (e.g. 5G EC Local
> Data Networks)  only optimize the ANYCAST traffic for the registered
> addresses.
>
> The network is already responsible for selecting the shortest path to one
> Application Load Balancer. draft-dunbar-lsr-5g-edge-compute-ospf-ext
> proposes to add additional weight in path selection.
>
>
>
> ANYCAST makes it possible to dynamically load balance across server
> locations based on network conditions. With multiple servers having the
> same ANYCAST address, it eliminates the single point of failure and
> bottleneck at the application layer load balancer that has the shortest
> routing distance. Another benefit of using ANYCAST address is removing th=
e
> dependency on how UEs get the IP addresses for their Applications. Some U=
Es
> (or clients) might use stale cached IP addresses for extended period.
>
>
>
> Network service providers can even offer this as a value added service,
> making network information more useful to deliver services to application=
s.
>
> Isn=E2=80=99t it a win-win approach for both network service providers an=
d the
> applications owners?
>
>
>
> As WG member,
>
>
>
> It's not a win when their network fails.
>
>
>
> At a high level I think the idea of a smart network is interesting. I
> don't have good initial feelings though about trying to achieve that by
> adding application load based metrics into the routing protocol. There's
> all sort of layer violations going on there for one, but perhaps more
> importantly, our routing protocols have not been tried and tested over th=
e
> decades with this use in mind.
>
>
>
> One could imagine designing a higher layer distributed load balancing
> application/protocol that utilized routing information though, something
> like that would align more closely with the layering we've been designing
> to all these years. It probably would not rely on anycast exclusively, bu=
t
> instead use anycast to talk to a server that implemented this LB protocol
> (something anycast is good at) which would provide a unicast address for
> the requested application, with the ability to adjust (reacquire a new
> unicast address, whatever) as conditions (either at the routing or
> application layer) change through notifications or polling. Just
> brainstorming here, but there are lots of ways one could imagine this
> working.
>
>
>
> Thanks,
>
> Chris.
>
>
>
>
>
> Linda Dunbar
>
>
>
> _______________________________________________
> Lsr mailing list
> Lsr@ietf.org
> https://www.ietf.org/mailman/listinfo/lsr
> <https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fwww.=
ietf.org%2Fmailman%2Flistinfo%2Flsr&data=3D04%7C01%7Clinda.dunbar%40futurew=
ei.com%7C090a578c2711455d431808d8e2c1ad0b%7C0fee8ff2a3b240189c753a1d5591fed=
c%7C1%7C0%7C637508668779231156%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi=
LCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3DeTZ9G6OwmE%2F=
HAJyIInOzMSSmQTmVcGFmXbrNm5YCMX4%3D&reserved=3D0>
>
> --
>
>
> <https://nam11.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fwww.v=
erizon.com%2F&data=3D04%7C01%7Clinda.dunbar%40futurewei.com%7C090a578c27114=
55d431808d8e2c1ad0b%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C6375086687=
79231156%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTi=
I6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3DeJ5%2ByOifw0UB7MvrjhqcBLWhvhO6n9QR3=
%2ByuuWKs2tk%3D&reserved=3D0>
>
> *Gyan Mishra*
>
> *Network Solutions Architect *
>
>
>
> *M 301 502-1347 13101 Columbia Pike
> <https://www.google.com/maps/search/13101+Columbia+Pike?entry=3Dgmail&sou=
rce=3Dg>
> *Silver Spring, MD
>
>
>
--=20

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *



*M 301 502-134713101 Columbia Pike *Silver Spring, MD

--00000000000093428c05bd3c57e3
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"auto">Hi Linda=C2=A0</div><div dir=3D"auto"><br></div><div dir=
=3D"auto">Comments in-line=C2=A0</div><div><br><div class=3D"gmail_quote"><=
div dir=3D"ltr" class=3D"gmail_attr">On Wed, Mar 10, 2021 at 6:46 PM Linda =
Dunbar &lt;<a href=3D"mailto:linda.dunbar@futurewei.com">linda.dunbar@futur=
ewei.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D=
"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;pad=
ding-left:1ex;border-left-color:rgb(204,204,204)">





<div lang=3D"EN-US" link=3D"blue" vlink=3D"purple" style=3D"word-wrap:break=
-word">
<div class=3D"m_6045252049156658787WordSection1">
<p class=3D"MsoNormal">Gyan, <u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">To a router, having multiple servers with the same (=
ANYCAST) address attached to different egress routers (A-ER) is same as hav=
ing multiple paths to reach the (ANYCAST) address.<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">You are absolutely correct that there are many tools=
 to influence the path section, such as the routing distance, TE metrics, p=
olicies, etc.
<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">draft-dunbar-lsr-5g-edge-compute-ospf-ext proposes t=
o add another component to influence the path selection: the =E2=80=9CSite-=
Cost=E2=80=9D =C2=A0which is influenced by =C2=A0=E2=80=9Csite-capacity + l=
oad measurement + Preference + xxx=E2=80=9D. The =E2=80=9Csite-Cost=E2=80=
=9D can be raw measurements
 collected by the egress routers based on the instruction from a controller=
, or informed by the App Controller periodically.
<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">In the past, ANYCAST has been predominantly used for=
 multiple servers in geographically far apart locations so that routing dis=
tance alone can always nail down to one specific ANYCAST server.
<u></u><u></u></p>
<p class=3D"MsoNormal" dir=3D"auto"><u></u>=C2=A0Gyan&gt; =C2=A0The Anycast=
 environments that I have worked with architecture have been server cluster=
s in data centers geographically diverse that sit behind a load balancer th=
at uses a concept called HRI host route injection that if the service is up=
 the VIP is BGP advertised for the cluster to DC core and then services VIP=
 host route are advertised into the core all BGP attributes equal so lowest=
 IGP metric tie breaker picks best path shortest path across the core =C2=
=A0as best path and of iBGP multipath is used that services VIPs are geogra=
phically load balanced flow based if metric is a tie and if IPv6 is used in=
 the core then 5-tuple per flow load balanced optimized. =C2=A0</p></div></=
div></blockquote><div dir=3D"auto"><br></div><div dir=3D"auto"><br></div><b=
lockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-le=
ft-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb=
(204,204,204)"><div lang=3D"EN-US" link=3D"blue" vlink=3D"purple" style=3D"=
word-wrap:break-word"><div class=3D"m_6045252049156658787WordSection1"><p c=
lass=3D"MsoNormal" dir=3D"auto"><u></u></p>
<p class=3D"MsoNormal">3GPP TR23.748 (5G Edge Computing) is proposing to us=
e multiple servers (or multiple App Layer Load Balancers) with the same ANY=
CAST address in their Local IP Data Network to avoid the single point of fa=
ilure and the bottleneck at the App
 Layer Load Balancer for mission critical applications. =C2=A0<u></u><u></u=
></p>
<p class=3D"MsoNormal" dir=3D"auto"><u></u>=C2=A0Gyan&gt; =C2=A0With Anycas=
t routing as I mentioned you don=E2=80=99t have a single point of failure a=
nd really have optimal redundancy which is why for any services such as DNS=
, NTP and many others Anycast is the best most redundancy and optimal as pr=
oximity routing is used. If your closest lowest IGP metric iBGP load balanc=
ed path goes down let=E2=80=99s say multiple DC outages you still have your=
 next closest in the BGP path list pecking order to reach the service thus =
optimized redundancy as every DC would have to be down for service VIP to b=
e down.=C2=A0 Also with Anycast as it a distributed architecture you are no=
t overloading core paths or certain DC VIPs as all traffic is distributed g=
eographically proximity load balanced automatically.=C2=A0 So there is a le=
sser chance of bottleneck as the Anycast architecture is distributed.</p></=
div></div></blockquote><div dir=3D"auto"><br></div><div dir=3D"auto">=C2=A0=
 =C2=A0As far as 5G you still have main components of the path from UE =C2=
=A0user data plane to RAN xHaul VPN within the wireless operator network th=
en handoff to the core to a service VIP in a closet proximity data center. =
So now we are trying to further optimize the cost based on real time PM per=
formance metrics to calculate the best path with this draft.</div><div dir=
=3D"auto"><br></div><div dir=3D"auto">This draft below is in LSR maybe inte=
resting to you related to flex algo bandwidth related constraints so that t=
he TE static ERO concept of exclude L2 links in a bundle can be accomplishe=
d based on link delay bandwidth thresholds that use semi dynamic metrics ba=
sed on PM measurements.</div><div dir=3D"auto"><br></div><div dir=3D"auto">=
In that discussion thread was brought up use of the PM based metrics and if=
 that would cause instability.=C2=A0 That maybe something to consider when =
using dynamic PM based metrics. =C2=A0</div><div dir=3D"auto"><br></div><di=
v dir=3D"auto"><div><a href=3D"https://datatracker.ietf.org/doc/html/draft-=
hegde-lsr-flex-algo-bw-con-01">https://datatracker.ietf.org/doc/html/draft-=
hegde-lsr-flex-algo-bw-con-01</a></div><div dir=3D"auto"><br></div><div dir=
=3D"auto">Kind Regards=C2=A0</div><div dir=3D"auto"><br></div><div dir=3D"a=
uto"><br></div><div dir=3D"auto">Gyan</div><br></div><blockquote class=3D"g=
mail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-=
left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div =
lang=3D"EN-US" link=3D"blue" vlink=3D"purple" style=3D"word-wrap:break-word=
"><div class=3D"m_6045252049156658787WordSection1"><p class=3D"MsoNormal" d=
ir=3D"auto"><u></u></p>
<p class=3D"MsoNormal">Thank you very much for your comments. I have made s=
ome changes to the text. Please see the revision:
<a href=3D"https://datatracker.ietf.org/doc/draft-dunbar-lsr-5g-edge-comput=
e-ospf-ext/" target=3D"_blank">
https://datatracker.ietf.org/doc/draft-dunbar-lsr-5g-edge-compute-ospf-ext/=
</a></p></div></div><div lang=3D"EN-US" link=3D"blue" vlink=3D"purple" styl=
e=3D"word-wrap:break-word"><div class=3D"m_6045252049156658787WordSection1"=
><p class=3D"MsoNormal"><u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">Linda<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<div style=3D"border-style:solid none none;border-top-width:1pt;padding:3pt=
 0in 0in;border-top-color:rgb(225,225,225)">
<p class=3D"MsoNormal"><b>From:</b> Gyan Mishra &lt;<a href=3D"mailto:hayab=
usagsm@gmail.com" target=3D"_blank">hayabusagsm@gmail.com</a>&gt; <br>
<b>Sent:</b> Tuesday, March 9, 2021 12:08 AM<br>
<b>To:</b> Liyizhou &lt;<a href=3D"mailto:liyizhou@huawei.com" target=3D"_b=
lank">liyizhou@huawei.com</a>&gt;<br>
<b>Cc:</b> Christian Hopps &lt;<a href=3D"mailto:chopps@chopps.org" target=
=3D"_blank">chopps@chopps.org</a>&gt;; Linda Dunbar &lt;<a href=3D"mailto:l=
inda.dunbar@futurewei.com" target=3D"_blank">linda.dunbar@futurewei.com</a>=
&gt;; <a href=3D"mailto:lsr@ietf.org" target=3D"_blank">lsr@ietf.org</a><br=
>
<b>Subject:</b> Re: [Lsr] Why not leverage Network conditions to optimize b=
alancing among multiple App Layer Load Balancers? as proposed by draft-dunb=
ar-lsr-5g-edge-compute-ospf-ext<u></u><u></u></p>
</div>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<div>
<div>
<p class=3D"MsoNormal">Linda and authors=C2=A0<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
</div>
<div>
<p class=3D"MsoNormal">Some thoughts regarding load balancing draft.<u></u>=
<u></u></p>
</div>
<div>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
</div>
<div>
<p class=3D"MsoNormal">Anycast in my experience has been used predominantly=
 in my experience within operators networks with BGP overlay, =C2=A0using B=
GP best path selection and most cases boils down to lowest IGP metric tie b=
reaker shortest path for the service Anycast
 proximity route which you can also with unique RD in overlay and can take =
advantage of iBGP multipath equal cost load balancing over an operator vacc=
ine or 4G/5G RAN xhaul or internet.<u></u><u></u></p>
</div>
</div>
<div>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
</div>
<div>
<p class=3D"MsoNormal">The nice thing about Anycast with BGP overlay you as=
 are automatically proximity based routing load balancing inherent to Anyca=
st routing.<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
</div>
<div>
<p class=3D"MsoNormal">Point here is we are using BGP best path selection b=
ut it does boils down to IGP lowest metric tie breaker but you can use iBGP=
 multipath to further optimize the routing for cloud computing.<u></u><u></=
u></p>
</div>
<div>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
</div>
<div>
<p class=3D"MsoNormal">We have so many tools in our operators toolbox to op=
timize routing SR or Flex-Algo, SDN etc am wondering if some form of SDN or=
 SD-WAN overlay could provide the Dyncast type Dynamic Anycast solution.<u>=
</u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
</div>
<div>
<p class=3D"MsoNormal">I want to the wiki page for Dyncast.=C2=A0 The prese=
ntation is not available yet.=C2=A0 Will check tomorrow.<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
</div>
<div>
<p class=3D"MsoNormal">Thanks=C2=A0<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
</div>
<div>
<p class=3D"MsoNormal">Gyan<u></u><u></u></p>
</div>
<div>
<div>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
</div>
<div>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<div>
<div>
<p class=3D"MsoNormal">On Mon, Mar 8, 2021 at 11:36 PM Liyizhou &lt;<a href=
=3D"mailto:liyizhou@huawei.com" target=3D"_blank">liyizhou@huawei.com</a>&g=
t; wrote:<u></u><u></u></p>
</div>
<blockquote style=3D"border-style:none none none solid;border-left-width:1p=
t;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in;border-left-co=
lor:rgb(204,204,204)">
<div>
<div>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">Hi,</span><u></u><u></u></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">=C2=A0</span><u></u><u></u></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">Sorry to chime in.</span><u></u><u></u></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">=C2=A0</span><u></u><u></u></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">There are certainly some higher layer application/protocols to employ. At=
 the same time, there are some advantages of network layer
 approaches as well in my mind.</span><u></u><u></u></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">=C2=A0</span><u></u><u></u></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">When talking about edge computing, there are some unique characteristics.=
 The number of edge sites could be large or huge in future
 in a big city. Edges are geographically scattered which could be a few, or=
 tens of, or a hundred kilometers away from each other, and each site has l=
imited computing resources which could be a small cluster. Application laye=
r based approach normally would
 rely on one or several =E2=80=9Cserver=E2=80=9D/=E2=80=9Dbroker=E2=80=9D t=
o be responsible for request handling all over the city. As such =E2=80=9Cs=
ervers=E2=80=9D are unlikely available on each and every edge site, it intr=
oduces additional path stretch for data packets requiring delivery to other=
 edge
 sites first. Such path stretch introduces additional (network and processi=
ng) delay which could be crucial for short live request flow. On the contra=
ry, the network node at the edge is naturally sitting on the data path with=
out introducing any additional cost
 to direct the (explicit/implicit) request somewhere else. Also routing sys=
tem has been proven doing good in such distributed manner.</span><u></u><u>=
</u></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">=C2=A0</span><u></u><u></u></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">=C2=A0</span><u></u><u></u></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">There is a dyncast (dynamic anycast) work ongoing. It is not exactly same=
 as what Linda proposed here, but some relations can be seen,
 like trying to use anycast methodology to access an edge computing, especi=
ally computational intensive, service. The current discussions are about co=
mpellingness of the use cases, the deficiency of existing solutions, and pr=
oposed architecture, not gone very
 far into what specific routing protocols to use yet. A side meeting will b=
e held on Wed 10am CET. You may check
<a href=3D"https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2=
F%2Fgithub.com%2Fdyncast%2Fietf110&amp;data=3D04%7C01%7Clinda.dunbar%40futu=
rewei.com%7C090a578c2711455d431808d8e2c1ad0b%7C0fee8ff2a3b240189c753a1d5591=
fedc%7C1%7C0%7C637508668779221152%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwM=
DAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=3D%2BARH=
REw51PPjAyjut9I9Z5Ym%2FV%2FDYwPl2xO2luZxIag%3D&amp;reserved=3D0" target=3D"=
_blank">
https://github.com/dyncast/ietf110</a> for more info. =C2=A0</span><u></u><=
u></u></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">=C2=A0</span><u></u><u></u></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">Cheers,</span><u></u><u></u></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">Yizhou</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class=3D"MsoNormal"><span style=3D"font-size:10.5pt;color:rgb(31,73,125)=
">=C2=A0</span><u></u><u></u></p>
<div>
<div style=3D"border-style:solid none none;border-top-width:1pt;padding:3pt=
 0in 0in;border-top-color:rgb(225,225,225)">
<p class=3D"MsoNormal"><b>From:</b> Lsr [mailto:<a href=3D"mailto:lsr-bounc=
es@ietf.org" target=3D"_blank">lsr-bounces@ietf.org</a>]
<b>On Behalf Of </b>Christian Hopps<br>
<b>Sent:</b> Tuesday, March 9, 2021 9:00 AM<br>
<b>To:</b> Linda Dunbar &lt;<a href=3D"mailto:linda.dunbar@futurewei.com" t=
arget=3D"_blank">linda.dunbar@futurewei.com</a>&gt;<br>
<b>Cc:</b> <a href=3D"mailto:lsr@ietf.org" target=3D"_blank">lsr@ietf.org</=
a>; Christian Hopps &lt;<a href=3D"mailto:chopps@chopps.org" target=3D"_bla=
nk">chopps@chopps.org</a>&gt;<br>
<b>Subject:</b> Re: [Lsr] Why not leverage Network conditions to optimize b=
alancing among multiple App Layer Load Balancers? as proposed by draft-dunb=
ar-lsr-5g-edge-compute-ospf-ext<u></u><u></u></p>
</div>
</div>
<p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>
<p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>
<div>
<p class=3D"MsoNormal" style=3D"margin-bottom:12pt"><u></u>=C2=A0<u></u></p=
>
<blockquote style=3D"margin-top:5pt;margin-bottom:5pt">
<div>
<p class=3D"MsoNormal">On Mar 8, 2021, at 7:40 PM, Linda Dunbar &lt;<a href=
=3D"mailto:linda.dunbar@futurewei.com" target=3D"_blank">linda.dunbar@futur=
ewei.com</a>&gt; wrote:<u></u><u></u></p>
</div>
<p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>
<div>
<div>
<p class=3D"MsoNormal">Christian,=C2=A0<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">You said at LSR session today that there might be co=
ncern of network optimizing ANYCAST traffic to better balance among multipl=
e App Layer Load Balancers.<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">First of all, only the Applications that need to lev=
erage the network condition to balance among their multiple Load Balancers =
will get the benefit of path selection that are based
 on the combination of routing distance and other dynamic running status. T=
he networks (e.g. 5G EC Local Data Networks) =C2=A0only optimize the ANYCAS=
T traffic for the registered addresses.<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">The network is already responsible for selecting the=
 shortest path to one Application Load Balancer. draft-dunbar-lsr-5g-edge-c=
ompute-ospf-ext proposes to add additional weight
 in path selection.<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">ANYCAST makes it possible to dynamically load balanc=
e across server locations based on network conditions. With multiple server=
s having the same ANYCAST address, it eliminates the
 single point of failure and bottleneck at the application layer load balan=
cer that has the shortest routing distance. Another benefit of using ANYCAS=
T address is removing the dependency on how UEs get the IP addresses for th=
eir Applications. Some UEs (or clients)
 might use stale cached IP addresses for extended period.<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">Network service providers can even offer this as a v=
alue added service, making network information more useful to deliver servi=
ces to applications.<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">Isn=E2=80=99t it a win-win approach for both network=
 service providers and the applications owners?<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">As WG member,<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>
</div>
<p class=3D"MsoNormal">It&#39;s not a win when their network fails.<u></u><=
u></u></p>
</div>
<div>
<p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">At a high level I think the idea of a smart network =
is interesting. I don&#39;t have good initial feelings though about trying =
to achieve that by adding application load based metrics
 into the routing protocol. There&#39;s all sort of layer violations going =
on there for one, but perhaps more importantly, our routing protocols have =
not been tried and tested over the decades with this use in mind.<u></u><u>=
</u></p>
</div>
<div>
<p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">One could imagine designing a higher layer distribut=
ed load balancing application/protocol that utilized routing information th=
ough, something like that would align more closely
 with the layering we&#39;ve been designing to all these years. It probably=
 would not rely on anycast exclusively, but instead use anycast to talk to =
a server that implemented this LB protocol (something anycast is good at) w=
hich would provide a unicast address
 for the requested application, with the ability to adjust (reacquire a new=
 unicast address, whatever) as conditions (either at the routing or applica=
tion layer) change through notifications or polling. Just brainstorming her=
e, but there are lots of ways one
 could imagine this working.<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">Chris.<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin-bottom:12pt"><u></u>=C2=A0<u></u></p=
>
<blockquote style=3D"margin-top:5pt;margin-bottom:5pt">
<div>
<div>
<p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal">Linda Dunbar<u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>
</div>
</div>
<p class=3D"MsoNormal">_______________________________________________<br>
Lsr mailing list<br>
<a href=3D"mailto:Lsr@ietf.org" target=3D"_blank">Lsr@ietf.org</a><br>
<a href=3D"https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2=
F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Flsr&amp;data=3D04%7C01%7Clinda.dunb=
ar%40futurewei.com%7C090a578c2711455d431808d8e2c1ad0b%7C0fee8ff2a3b240189c7=
53a1d5591fedc%7C1%7C0%7C637508668779231156%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi=
MC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=
=3DeTZ9G6OwmE%2FHAJyIInOzMSSmQTmVcGFmXbrNm5YCMX4%3D&amp;reserved=3D0" targe=
t=3D"_blank">https://www.ietf.org/mailman/listinfo/lsr</a><u></u><u></u></p=
>
</blockquote>
</div>
</div>
</div>
<p class=3D"MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p><a href=3D"https://nam11.safelinks.protection.outlook.com/?url=3Dhttp%3A=
%2F%2Fwww.verizon.com%2F&amp;data=3D04%7C01%7Clinda.dunbar%40futurewei.com%=
7C090a578c2711455d431808d8e2c1ad0b%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7=
C0%7C637508668779231156%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjo=
iV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=3DeJ5%2ByOifw0UB7M=
vrjhqcBLWhvhO6n9QR3%2ByuuWKs2tk%3D&amp;reserved=3D0" target=3D"_blank"><spa=
n style=3D"text-decoration:none;color:rgb(17,85,204)"><img border=3D"0" wid=
th=3D"81" height=3D"18" style=3D"width: 0.8437in; height: 0.1875in;" id=3D"=
m_6045252049156658787_x0000_i1025" src=3D"http://ss7.vzw.com/is/image/Veriz=
onWireless/vz-logo-email"></span></a><span style=3D"color:rgb(34,34,34)"><u=
></u><u></u></span></p>
<p style=3D"margin:0in"><b><span style=3D"font-family:Arial,sans-serif;colo=
r:black">Gyan Mishra</span></b><span style=3D"font-family:Arial,sans-serif;=
color:black"><u style=3D"font-family:Arial,sans-serif"></u><u style=3D"font=
-family:Arial,sans-serif"></u></span></p>
<p style=3D"margin:0in"><i><span style=3D"font-family:Georgia,serif;color:b=
lack">Network Solutions Architect=C2=A0</span></i><span style=3D"color:rgb(=
34,34,34)"><u></u><u></u></span></p>
<p style=3D"margin:0in"><i><span style=3D"font-family:Georgia,serif;color:b=
lack">M 301 502-1347<br>
<a href=3D"https://www.google.com/maps/search/13101+Columbia+Pike?entry=3Dg=
mail&amp;source=3Dg" style=3D"font-family:Georgia,serif">13101 Columbia Pik=
e</a>=C2=A0<br>
</span></i><span style=3D"color:black">Silver Spring, MD<u></u><u></u></spa=
n></p>
</div>
<div>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>

</blockquote></div></div>-- <br><div dir=3D"ltr" class=3D"gmail_signature" =
data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div dir=3D"ltr"><div d=
ir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"l=
tr"><div><p style=3D"color:rgb(34,34,34)"><a href=3D"http://www.verizon.com=
/" style=3D"color:rgb(17,85,204);padding-bottom:1em;display:inline-block" t=
arget=3D"_blank"><img src=3D"http://ss7.vzw.com/is/image/VerizonWireless/vz=
-logo-email" width=3D"81" height=3D"18" style=3D"height:18px;width:81px"></=
a><br></p><p style=3D"font-size:1em;margin:0px;font-family:&quot;Verizon NH=
G DS&quot;,Arial,sans-serif;line-height:13px;color:black"><b>Gyan Mishra</b=
></p><p style=3D"color:rgb(34,34,34);margin:0px;line-height:13px"><font fac=
e=3D"georgia, serif" style=3D"color:black;font-size:1em"><i>Network Solutio=
ns A</i></font><font color=3D"#000000" face=3D"georgia, serif"><i>rchitect=
=C2=A0</i></font></p><p style=3D"font-size:1em;margin:0px;line-height:13px;=
color:black"><i><font face=3D"georgia, serif">M 301 502-1347<br>13101 Colum=
bia Pike=C2=A0<br></font></i>Silver Spring, MD</p></div><div><br></div></di=
v></div></div></div></div></div></div></div>

--00000000000093428c05bd3c57e3--

