Re: [GROW] comments on draft-lapukhov-bgp-routing-large-dc-06

[Nick Hilliard <[e-mail]>]

On 06/09/2013 03:45, Jon Mitchell wrote:
> We would like to solicit any further comments on
> draft-lapukhov-bgp-routing-large-dc-06.  Originally this draft was
> presented by Petr in Vancouver in both IDR and GROW and we feel this
> draft is useful to the IETF community as it documents a working
> scalable design for building large DC's.

i think this draft is really interesting.  A couple of things come to mind:

- default route origination is a real pain in larger scale networks for
obvious reasons, not least because you can often end up with non-optimal
egress choice (e.g. the RR's choice of default route).  You also need to
take special care to ensure that the scope of the default route is limited
to only router sets that actually need it.  I prefer the idea of using a
floating anycast default route.  i.e. you inject an anycast prefix from all
or a selection of all routers which can handle default free traffic flow,
then on small-fib routers you can install a static default pointing to this
floating default and depend on recursive route lookup to figure out the
best path (original idea from Saku Ytti - http://goo.gl/Nj69OZ).  This
allows much tighter control of default route propagation, with better
egress choice characteristics.

- in virtualised networks hosting third party tenancies, it is iften useful
to extend L3 to the hypervisor.  With current tech, running thousands of
vms per cabinet is not unrealistic, and this number will undoubtedly
increase over time.  This brings up the issue of both how to handle address
assignment in an optimal manner and also how to be able to assign one or
more IP addresses per vm or vm cluster, without the problems associated
with flat shared lans, without using NAT and its consequent problems, but
also without wasting precious public IP addresses on link networks which
typically have abysmal assignment efficiency (e.g. 50% for /30, 75% for
/29, etc).  There are models out there which suggest using routing
protocols, but BGP may not always be the best choice due to scalability
issues: if you have a ToR switch, it will be pretty limited in terms of the
number of bgp sessions it can handle.  Some people have approached this by
using RIP to the client VMs.  The advantage of this is that RIP is fully
stateless, whereas BGP has session overhead.  On a small ToR box with an
itty bitty RP, the overhead associated with high hundreds hundreds or even
low thousands of BGP sessions may be too much.  I'm not sure if this is in
the scope of this draft though.

- i think you skimp over the problems associated with bgp session failure
detection / reconvergence.  Mandating ebgp will get rid of the problems
associated with the traditional loopback-to-loopback configuration of most
ibgp networks, but there are still situations where dead link detection is
going to be necessary using some form of keepalive mechanism which works
faster than ebgp timers.  It would be good to have a little more discussion
about bfd - if we can point vendors to good use cases here, they will be
more inclined to support bfd on tor boxes.

- typo: s/it's/its/g

Nick