Re: [dc] Elevator Pitch

["Ashish Dalela (adalela)" <[e-mail]>]

100K or whatever that number is, is a big deal at the access. And that number will increase with time. The thing to bear in mind is that these are ADDITIONAL entries, not in lieu of other entries. E.g. network routes, local host-port entries, ACLs etc exist today and will continue to exist. Host routes are in addition to that. Access switches generally had about 16K or 32K table sizes. And we are talking about adding 3-4 times more to that.

The other thing is that access to core ratio is about 1000:1. So, we need to multiply the incremental costs by 1000 to get the full impact to TCO.

In regard to scaling, there are 4 places to look at:

1. Access
2. Core
3. Inter-datacenter
4. Datacenter-internet

These have different scaling properties. 

If you have different DC and DCI technologies, and you use Encap, you have high entries at Access, DCI and Datacenter-Internet. 

If you have common DC and DCI technologies, you have high Access and Datacenter-Internet entries.

If you have hierarchical MAC, then you have only high in Datacenter-Internet boundary. 

Net-net, out of the 4 things above, you can solve 3 (first 3). The fourth we have to distribute over multiple routers, which is not hard because it is always north-south traffic. 

Thanks, Ashish


-----Original Message-----
From: [e-mail] [mailto:[e-mail]] On Behalf Of Christopher LILJENSTOLPE
Sent: Thursday, December 29, 2011 10:58 PM
To: Russ White
Cc: [e-mail]
Subject: Re: [dc] Elevator Pitch

Greetings Russ et.al.,

On 29Dec2011, at 07.08, Russ White wrote:

> 
>> Actually, 100K host routes in leaf nodes may well be a Big Deal. It
>> depends on what type of devices need to understand host routes.
> 
> So let me try to abstract the problem a little... There are two things
> that cost money in building a network:
> 
> 1. State --the more you carry, the more hardware is going to cost.
> 2. Bandwidth --the less optimal your bandwidth utilization, the more
> you're spending on something that's not used.
> 
> The first rule of all network design is that the more state you put in a
> network, the more optimal your bandwidth usage. To reduce the cost of
> bandwidth, you have to spend money on state. To reduce the cost of
> state, you must increase your spending on bandwidth.

Agreed, mostly.  The question is where the state resides, control plane or data plane (do you differentiate between possible state and active state).  Control plane "possible state" should be cheaper (by orders of magnitude) than data plane "active state" - however, transitioning between the two is a complex problem (that has gotten somewhat easier as we have gotten smarter about distributed systems over time)
> 
> The second of network design is the balance point where the network is
> cheapest is different in every network.

agreed

> 
> So, IMHO, the point of protocol design should be to allow the design the
> maximum amount of flexibility and granularity in when and where to hide
> information, to make the right tradeoff for any particular network.
> 
agreed

> Now, to return to the discussion at hand: If you don't want to pay for
> devices that will handle 100 million routes, but you have 100 million
> devices, then you're going to have some degree of suboptimal bandwidth
> utilization. There's simply no way around this reality.

agreed
> 
> IMHO, however we resolve this problem, we need to resolve it in a way
> that allows an operator to support 100 million routes in every node, if
> they choose to optimize for bandwidth at the cost of hardware. Or to
> support 1 route in every node using really cheap hardware, if they want
> to optimize for state. Or for any point in the sliding scale in between
> (within reason).

agreed

> 
> So this control plane needs to:
> 
> 1. Be able to support 100 million destinations at the host level (that
> doesn't mean 100 million paths, but only 100 million destinations --two
> different problems).

yes

> 
> 2. Be able to aggregate to hide information at anyplace that's logical
> within the network.

yes

> 
> BTW, some folks would like to solve this problem by making the control
> plane react to the data plane --but this carries it's own baggage in
> complexity and in operational capabilities. Reactive control planes
> always converge more slowly, and waste bandwidth at a rate that's
> arguably higher than simple aggregation. So there's no "silver bullet"
> waiting in the wings in the form of caching at the control plane level.

however, I believe that aggregation time may be coming down, especially in a more homogenous environment (like a DC) vs a heterogeneous environment (like the DFZ).

> 
>>> 2. Why does this mobility need to be at layer 2 specifically? Are we
>>> assuming DDNS and other sorts of solutions in this space will simply
>>> never be fast enough/scale far enough/etc?
>> 
>> Like it or not, the key requirement for VM mobility is that the VM's
>> IP address does not change. That means the VM can't really move from
>> one IP subnet to another. That means either moving to bigger and
>> bigger L2s (all under one IP subnet) as the DC expands or the need to
>> inject /32 host routes.
> 
> Again, the same tradeoff as above --moving to a bigger l2 domain also
> means losing the ability to optimally direct traffic through the network
> (unless you put another control plane on top of the l2 and l3 control
> planes already in existence --which just adds the complexity you're
> trying to get away from in the host routes back into the network state!).

Or replacing one or two of those L2/L3 control planes with a control plane with a global view, which is easier to do in a constrained network like a traffic engineering core or a data center.  It's is a bounded problem.  As you stated, however, convergence MAY be slower, but I would argue that converging a network of Nx10K switching/routing nodes takes a non-trivial amount of time as well :)

> 
>> Neither of those approaches seems particularly scalable/desirable if
>> you look 10 years down the road and think of 1M+ physical machines in
>> a DC.
> 
> There is no "particularly desirable" solution, in reality, because
> building a network with no control plane state anyplace that uses
> bandwidth in a perfectly optimal way is simply impossible no matter how
> you slice it (unless you're going to go into quantum routing!).
> 
> :-)
> 
> Russ
> _______________________________________________
> dc mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/dc

--  
李柯睿
Check my PGP key here: https://www.asgaard.org/~cdl/cdl.asc
Current vCard here: https://www.asgaard.org/~cdl/cdl.vcf

_______________________________________________
dc mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/dc