Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion?

[Igor Gashinsky <[e-mail]>]

On Wed, 15 Feb 2012, Dino Farinacci wrote:

:: > I've so far stayed pretty quiet on this, but, on this, I have to strongly 
:: > disagree. It is not FUD that multicast doesn't scale well inside large 
:: 
:: There are others who probably wish to not divulge their proprietary 
:: scalign numbers who would disagree with you.

Would anybody be willing to speak about orders of magnitude here? We had 
to do a *lot* of nasty, ugly, hacky-ish things to get mcast to scale to 
the 10's of Ks (S,G) scale, but, when we looked at what it would take to 
make the next order-of-magnitude jump to 100k+, that's when the 
cost/complexity tradeoff simply made it not practical -- I'd love to hear 
if anybody is actually doing 100k+ (S,G) of random-ish traffic profile.

:: > datacenters -- it is a simple fact, and I speak as an operator of what 
:: > several of my vendors called the 2nd largest multicast deployment they 
:: > have ever seen, with many 10's of thousands (S,G) entries. 
:: 
:: It depends what you are comparing 10,000 to. Comparing to unicast 
:: numbers would not being comparing apples with apples. Remember the 
:: granularity of a multicast route is much finer than a unciast route 
:: because it wants to conserve bandwidith and build good distribution 
:: trees.
:: 
:: It is a simple bandwidth versus state tradeoff and 10,000 is pretty large.

You are absolutely right.. but the tradeoff is mostly dependant on the 
*degree of replication* for all those mcast routes vs the state that they 
cost. For my network, the *average* degree of replication is actually 
quite small (on the order of 2-3 hosts per group, taking up a tiny 
fraction of the bandwidth), so trading that amount of bandwidth for state 
was a no brainer. However, if you have very few groups with a very high 
replication factor (say, 1000 hosts), then you would arrive at a very 
different conclusion, but, then you likely don't have a multicast scaling 
problem, since then you likely only have a few hundred (S,G) entries.. So, 
do people have some other datapoints on degree of replication, amount of 
bw saved by mcast vs state on large datacenter (ie not-designed-for-video) 
multicast networks?

:: Many think a data center with 10,000 hosts are large too. I know you 
:: can one-mag-up us Igor, but 10,000 is a large number for enterprise 
:: sites.

I guess that brings us to the crux of the question -- ARMD = Address 
Resolution for *Massive* numbers of hosts in the Data center (from the 
charter), so, what is the order of magnitude for massive? :)

To me, (and perhaps i'm in a very small minority here?), massive implies 
cloud scale datacenters, so, 10-20k physical hosts *per cluster*, and 
somewhere around 400-500k VM's is the absolute *minimum* for what I would 
concider to be massive (and, really, I think 100k physical, 2M+ logical is 
what i believe a realistic aiming point these days should be). 

If that's the case, then we are looking at about 2-order-of-magnitude 
higher then what most enterprises do.. so, what problem space do we want 
to solve? 

Thanks,
-igor

PS as you can probably tell, Dino and I have had this discussion before, 
quite a few times :)

--------------------+----------------------+------------------
   Igor Gashinsky   | Network Architecture | Yahoo! Inc.
 [e-mail] |  cell 917.807.2213   | Do You... Yahoo?
--------------------+----------------------+------------------