Re: [dc] new drafts

Robert,

Here are some things to evaluate scalability against.

Assume a simple case that under a switch there are 250 VM, split amongst
10 customers. Each customer has a unique VRF. Normally, we would have
advertized a /24 route for that switch. In this case your routes to a
single switch are segmented and there are 10 VRFs, and you will very
likely have 250 route table entries total segmented by VRF-ids. That's a
routing table bloat from 1 entry to 250 entry. This happens everywhere.
I have assumed a public IP addressing, but the same thing will happen
for the private addressing as well.

Then, typically the number of VRFs you can support on a router is about
4K. These # of VRFs have to be supported at the access, so you have to
assume this is the limit from the access viewpoint. 4K is nothing - we
have 4K VLANs today to segment and that's nothing. Every segmentation
technique being talked about speaks of a million plus segments. Take
that to VRFs, you need a million VRFs on the control plane at the access
switch. Another problem with a VRF is that it will get and store a route
for a host, even when there is no host talking to it. With dynamic
learning or learning based on packet arrival you avoid these host routes
and limit them to active conversations only. That's a huge saving
because not every host talks to every host.

Then, at massive scale, the failure rates are also massive. At 5 nines
reliability, a hardware entity out of 100,000 will fail every 5.25
minutes. Access switches don't have high availability. Software fails
even faster - OS is generally 4 9's, which means one out of 10,000 fails
every 5.25 minutes. At millions of instances of such entities, there are
rapid failures happening. You have to only look at massive datacenters
today run by Web 2.0 companies, and they all echo this view. They
basically form clusters of the same application. Software moves the
workload from one cluster to another. The whole cluster can fail over.
That's not what you do in a consumer cloud, where you have to recover.
At massive failure rates, and rapid recovery rates, you are moving
things around and injecting host routes for reachability. It's a
convergence problem, especially with link-state algorithms.

If the VM can be moved, then all you need to do is install a temporary
redirect of packets to the new location. Each host will refresh the MAC
after 15-30 seconds. If the packets are redirected from old to new
location for these 30 seconds, the redirect can be aged automatically.
This happens all the time in mobile networks in what is called a "fast
handoff" where you redirect the packets until handoff is completed.

Thanks, Ashish

-----Original Message-----
From: Robert Raszuk [mailto:robert@raszuk.net] 
Sent: Tuesday, January 03, 2012 9:24 PM
To: Ashish Dalela (adalela)
Cc: Yakov Rekhter; Aldrin Isaac; dc@ietf.org
Subject: Re: [dc] new drafts

Ashish,

VM mobility based on re-propagating host routes even with your own 
domain looks to me like broken .. VM mobility timing requirements are in

milliseconds or 10s of ms .. routing distribution or even FIB update 
with host routes (assuming that we could propagate host routes ahead of 
migration then just trigger the activation) would be in 100s of ms or 
seconds depending on the scope. I agree with your assertion that this is

not scalable.

However this has nothing to do with stating that VRFs are "just not 
scalable". VRF instantiation or VRF scaling is very opaque topic. 
Especially one could claim that VRFs in control plane are a very 
scalable tools we have at our disposal today.

Best,
R.

> Yakov,
>
>>> Would you care to provide detailed technical analysis to support
your
> claim about VRFs "just not scalable",
>
> VM mobility requires insertion of host-routes. As VMs move and/or are
> created/deleted, these host routes need to be propagated everywhere.
> That's a control plane scaling problem, plus a problem about
> convergence.
>
> Thanks, Ashish
>
>
> -----Original Message-----
> From: dc-bounces@ietf.org [mailto:dc-bounces@ietf.org] On Behalf Of
> Yakov Rekhter
> Sent: Tuesday, January 03, 2012 8:40 PM
> To: Ashish Dalela (adalela)
> Cc: dc@ietf.org; Aldrin Isaac
> Subject: Re: [dc] new drafts
>
> Ashish,
>
>> Hi Aldrin,
>>
>>>> How would you address gratuitous ARP when using the hierarchical
> MAC
>> addressing with a registry to store MAC-IP bindings?
>>
>> We should ideally have a protocol that maps ARP messages into the
>> registry and vice versa. Why new protocol? Greater security and
>> reliability plus avoiding ARP broadcasts. Gratuitous ARP doesn't have
> an
>> acknowledge, so if you want to perform some actions based on this,
you
>> need to get an "acknowledge" for it. Gratuitous ARP can get
> dropped.=20
>>
>> The other issue is that a user can misuse it do MAC hijacking. But if
>> you have a hierarchical MAC you can't hijack because network knows
> your
>> location. Things like dot1x which solved these issues in the campus
>> space are not used in datacenter.=20
>>
>>>> Also, when using MAC prefixes in complex L2VPN topologies how could
>> we address the risk of a hub forwarding into the wrong spoke context
> on
>> the leaf switch?  Each egress port on the leaf switch may be in a
>> different context (ex: EVI in EVPN).
>>
>> We should avoid building any kind of VRF at the control plane,
because
>> that's just not scalable.
>
> Would you care to provide detailed technical analysis to support your
> claim about VRFs "just not scalable", or should this claim be treated
> as proof by emphatic assertion ?
>
> Yakov.
> _______________________________________________
> dc mailing list
> dc@ietf.org
> https://www.ietf.org/mailman/listinfo/dc
> _______________________________________________
> dc mailing list
> dc@ietf.org
> https://www.ietf.org/mailman/listinfo/dc
>
>