Re: [dc] [armd] IP over IP solution for data center interconnect

[<[e-mail]>]

Just saw this:

> If the VM can be moved, then all you need to do is install a temporary
> redirect of packets to the new location. Each host will refresh the MAC
> after 15-30 seconds. If the packets are redirected from old to new
> location for these 30 seconds, the redirect can be aged automatically.
> This happens all the time in mobile networks in what is called a "fast
> handoff" where you redirect the packets until handoff is completed.

"all you need to do" should incur a $5 fine for the network version of
"a simple matter of programming" :-).

I see a bunch of problems here:

Mobile networks aren't running VMs.  ARP caches matter - a little bit of
Googling turns up multiple minute timeouts for some older OS versions and 4
hours for some routers.

In addition, some VMs need static MACs (e.g., software license key or
encryption key may be bound to the MAC).  There will be other reasons why
a VM's MAC cannot be changed when it is live migrated.

This also doesn't cover physical host usage of the burnt-in hardware MAC
on the NIC.  I assume that those need to continue to work as-is.  For
46-bits, I'm guessing that the Hierarchical MAC approach takes all the
locally administered MACs.  If any of those MACs are already in use,
that'll be a problem.

It also looks like it's seriously disruptive to existing L2 network management.

I see no mention of TRILL, SPB or PBB in the draft that advocates Hierarchical
MACs - those are three existing approaches to this problem that can aggregate
without forcing MAC changes on live VM migration.  At the very least, a solid
comparison to existing technology is in order.

Finally, this sort of MAC hierarchy work probably belongs in IEEE, not IETF.

Thanks,
--David


> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On Behalf Of [e-mail]
> Sent: Tuesday, January 03, 2012 11:42 AM
> To: [e-mail]
> Cc: [e-mail]
> Subject: Re: [dc] [armd] IP over IP solution for data center interconnect
> 
> Ashish,
> 
> > > [AD] The higher bits identify a switch - it's a switch-id.
> 
> That breaks VM migration across switches by forcing a MAC change.
> 
> Thanks,
> --David
> 
> > -----Original Message-----
> > From: [e-mail] [mailto:[e-mail]] On Behalf Of Ashish Dalela (adalela)
> > Sent: Tuesday, January 03, 2012 11:15 AM
> > To: [e-mail]
> > Cc: Pedro Marques; [e-mail]
> > Subject: Re: [dc] [armd] IP over IP solution for data center interconnect
> >
> > Robert,
> >
> > Please see inline.
> >
> > -----Original Message-----
> > From: Robert Raszuk [mailto:[e-mail]]
> > Sent: Tuesday, January 03, 2012 8:24 PM
> > To: Ashish Dalela (adalela)
> > Cc: Pedro Marques; [e-mail]
> > Subject: Re: [dc] [armd] IP over IP solution for data center
> > interconnect
> >
> > Ashish,
> >
> > OK let's just discuss what is in your draft on Hierarchical Addressing.
> >
> > 1. You have 48 bits 32 go for host remaining 16 goes for switches. How
> > do you aggregate at the TOR or AGGR switch boundary ? Are you assuming
> > single HOST - SWITCH with max 65K flat macs ?
> >
> > [AD] The higher bits identify a switch - it's a switch-id. The hosts are
> > dynamically assigned a host-id under that switch. Let's assume 23 bits
> > are for switch-id and 23 bits for host-id. To forward a packet to the
> > host, you only have to look at the first 23 bits. That's a MAC prefix to
> > route against.
> >
> > [AD] You can have 2^23 switches in a network and 2^23 hosts under each
> > switch.
> >
> > 2. Can you deploy this on existing VMs and existing switches ?
> >
> > [AD] What do you mean by this? Any VM can be configured with any MAC.
> > Any physical host can be configured with any MAC on any logical
> > interface. Configuration standpoint this is possible. Forwarding
> > standpoint, that's another question.
> >
> > 3. What new protocol you envision to use to distribute those new MACs ?
> >
> > [AD] IS-IS extensions. It can be TRILL extensions.
> >
> > 4. What is the advantage of using this vs ILNP if we assume that hosts
> > should be modified ?
> >
> > [AD] I'm not familiar with the ILNP work, but I'm assuming you are
> > talking about Loc-Id separation. If not, correct me. If yes, each Loc-Id
> > binding can be a host route, with mobility. These host-routes are a
> > scaling problem. Traditional IP packet have IP as ID and MAC as LOC. We
> > are just extending this LOC to make it actually location aware rather
> > than a flat address which is fixed regardless of where the location is.
> >
> > 5. The proposal does not support aggregation .. even the draft says it
> > :)
> >
> > "The total number of hardware entries anywhere in the network equals the
> >
> > total number of switches and remains agnostic of VM mobility."
> >
> > [AD] For 1 million VM, and 25 VM per host, you need 40,000 hosts. With
> > 48 port access switches, you need 833 switches. That's the routing table
> > size for any switch in the datacenter - core, aggregation, access.
> > Contrast this with host-routes, if each VM talks to 100 VMs, then each
> > access switch needs 48 * 25 * 100 = 120,000 host routes. Just because
> > the network prefix is 23 bits does not mean we have to store 10^23
> > prefixes. We have to store only as many switches as there are in the
> > network. Ratio between VM : switch is 1000 : 1 (today, assuming 48 port
> > access and 20 VM per port). That means instead of storing host-routes
> > which will grow proportional to VM growth, we store switch-id, which
> > will grow at 1000 times slower rate. As VM density increases, this
> > growth rate is further slowed down. There are other techniques to
> > further reduce the rate of growth. But in any case, 1000 times slower is
> > a lot slow.
> >
> > So if I have 100K switches I can not do any aggregation and need to
> > "route" 100K MAC addresses.
> >
> > [AD] I don't know how you came to that conclusion. Think of HMAC as an
> > IP address. Instead of 32 bits it is 46 bits. You route by prefixes in
> > L3, and you are routing by the same prefixes here. Just as you aggregate
> > IP, same way you aggregate MAC. It's not different.
> >
> > 6. Who provides me the mapping between switch mac and host/vm mac behind
> >
> > such switch ? Do switches proxy arp globally within your domain ?
> >
> > [AD] Variation of the same question. Above should answer it.
> >
> > Thx,
> > R.
> >
> >
> > > Robert,
> > >
> > >>> So you are advocating solution which is based on encapsulation -
> > that
> > > is fine.
> > >
> > > No, I'm not. Did you read the draft I had mentioned?
> > > Hierarchical MAC is not encapsulation. It is one 48 bit address.
> > >
> > >>> However how could you ever arrive at the conclusion that HMACs would
> > >>> scale better then "anything we know". Well I don't know about you,
> > > but I
> > >>> know that the key to scaling is ability to aggregate. And it is not
> > > that
> > >>> huge mystery that MACs aggregate rather poorly while there are quite
> > >>> well deployed protocols (be it IPv4 or IPv6) which aggregate
> > natively
> > >
> > > You are hitting the issue on the nail. So, read the draft I mentioned.
> > > Hierarchical MAC is higher bits "network prefix" and lower bits "host
> > > id".
> > > That's summarizable like IP address and aggregated.
> > > It has 46 bits to modify so larger than IPv4 internet.
> > >
> > > I won't comment on the rest, because you have made an assumption about
> > > encapsulation.
> > >
> > > I refer to this -
> > > http://tools.ietf.org/html/draft-dalela-dc-approaches-00.
> > >
> > > Thanks, Ashish
> > >
> > >
> > > -----Original Message-----
> > > From: Robert Raszuk [mailto:[e-mail]]
> > > Sent: Tuesday, January 03, 2012 7:05 PM
> > > To: Ashish Dalela (adalela)
> > > Cc: Pedro Marques; [e-mail]
> > > Subject: Re: [dc] [armd] IP over IP solution for data center
> > > interconnect
> > >
> > > Ashish,
> > >
> > >> The issues of scale you mentioned don't exist in Hierarchical MACs,
> > >> which scale better than anything we know of.
> > >
> > > So you are advocating solution which is based on encapsulation - that
> > is
> > > fine.
> > >
> > > However how could you ever arrive at the conclusion that HMACs would
> > > scale better then "anything we know". Well I don't know about you, but
> > I
> > > know that the key to scaling is ability to aggregate. And it is not
> > that
> > >
> > > huge mystery that MACs aggregate rather poorly while there are quite
> > > well deployed protocols (be it IPv4 or IPv6) which aggregate natively.
> > >
> > > For inter-dc this is IMHO a must. A must even if you build it using
> > > traditional routers or OF enabled switches - does not matter.
> > >
> > >> I don't want to split the requirements into multiple use-cases
> > >> because then this DC group will be many groups - one doing L2 and
> > >> another doing L3. That I think you will agree is not optimal for
> > >> anyone
> > >
> > > Why MAC-in-IP does not solve it for everyone ? And there are deployed
> > > solutions already ..
> > >
> > > IMHO what this group should accomplish is not to try to reinvent the
> > > world, but perhaps as example discuss where is the right boundary of
> > > encapsulation, how should we communicate between network and hosts,
> > what
> > >
> > > kind of DC instrumentation should be IETF blessed for easy integration
> > > (ie min subset of functionality it should possess etc .... )
> > >
> > > R.
> > >
> > > _______________________________________________
> > > dc mailing list
> > > [e-mail]
> > > https://www.ietf.org/mailman/listinfo/dc
> > >
> > >
> >
> > _______________________________________________
> > dc mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/dc
> 
> _______________________________________________
> dc mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/dc