Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion?

[Igor Gashinsky <[e-mail]>]

On Tue, 14 Feb 2012, Mike McBride wrote:

:: Anoop,
:: 
:: On Tue, Feb 14, 2012 at 12:32 PM, Anoop Ghanwani <[e-mail]> wrote:
:: > The whole problem of sending L2 multicast over a campus or data center
:: > backbone, in any sort of significant way, is a new one enabled for the first
:: > time by overlays.  There are interesting challenges when pushing large
:: > amounts of multicast traffic through a network, and  have thus far been dealt
:: > with using purpose-built networks.  While the overlay proposals have
:: > been careful not to impose new protocol requirements, they have not
:: > addressed the issues of performance and scalability, nor the large-scale
:: > availability of these protocols.
:: 
:: There are interesting challenges with unicast and multicast in big
:: flat DC networks. That's why we are here. One of the constant themes
:: we hear with multicast is the FUD that it doesn't scale. Especially
:: from vendors who are pushing another solution or who have a crappy
:: multicast implementation. Meanwhile multicast is being deployed in
:: large scale networks without scaling issue. Large L2 overlay networks?
:: I don't know. Would be good to find out from the community about
:: performance and scalability of multicast in the DC.

I've so far stayed pretty quiet on this, but, on this, I have to strongly 
disagree. It is not FUD that multicast doesn't scale well inside large 
datacenters -- it is a simple fact, and I speak as an operator of what 
several of my vendors called the 2nd largest multicast deployment they 
have ever seen, with many 10's of thousands (S,G) entries. 

Sorry, but many of us who operate really large scale datacenters with 
a lot of variability in the communication flows have removed all 
*application* use of multicast from the network due to scalability 
and stability reasons (ie by the time an (S,G) entry is programmed into 
hardware, we might be 3k+ packets deep into the flow, or the flow is over 
already, or what happens when you need to scale to 20k+ (S,G)'s?). 
Obviously, network protocols are still able to use multicast, but due 
to the very low volumes of flows and (S,G)s that is still acceptable.

Yes, we could have pushed the multicast scaling capabilities a bit higher 
by use of BIDIR, and yes, if the network hardware/software was built 
better it could have scaled a bit further, but, that's not the world we 
live in, and that would have cost significant troubleshooting capability, and a 
lot of extra complexity which we chose to instead solve via unicast, more 
bandwidth, and better application software.

So, whatever overlay protocols are proposed, I would suggest that counting 
on there being multicast underneath to transport you is not a very good 
idea...

Thanks,
-igor