From: <david.black@emc.com>
To: <adalela@cisco.com>
Date: Tue, 3 Jan 2012 12:02:37 -0500
Thread-Topic: [dc] [armd] IP over IP solution for data center interconnect
Thread-Index: AczKJ4svx6tIhjxTSoCEVFtst/+CNAAB77UAAAHJgoAAABTxIA==
Message-ID: <7C4DFCE962635144B8FAE8CA11D0BF1E05A5E7BBC5@MX14A.corp.emc.com>
References: <AFA7E5B6-4ABE-46FA-95B2-80BC5D3F62DA@netapp.com><1FEE3F8F5CCDE64C9A8E8F4AD27C19EE762EB4@szxeml525-mbs.china.huawei.com><618BE8B40039924EB9AED233D4A09C5102B250DA@XMB-BGL-416.cisco.com><4EFC947A.4020007@riw.us><7C4DFCE962635144B8FAE8CA11D0BF1E05A5CC9B4B@MX14A.corp.emc.com><618BE8B40039924EB9AED233D4A09C5102B25360@XMB-BGL-416.cisco.com><CAMXVrt6Spg9LVrwUoLm3XPGFt-stNqrKsO+SE6QsN4raEb1sLw@mail.gmail.com><618BE8B40039924EB9AED233D4A09C5102B2538D@XMB-BGL-416.cisco.com>
 <CAMXVrt4X+wB4AncQFpyt3H=iVg14rc_gHN8uF6D8n4XdAyBf1Q@mail.gmail.com>
 <618BE8B40039924EB9AED233D4A09C5102B25647@XMB-BGL-416.cisco.com>
 <4F030418.1070202@raszuk.net>
 <618BE8B40039924EB9AED233D4A09C5102B25683@XMB-BGL-416.cisco.com>
 <4F031689.1050303@raszuk.net>
 <618BE8B40039924EB9AED233D4A09C5102B256A3@XMB-BGL-416.cisco.com>
 <7C4DFCE962635144B8FAE8CA11D0BF1E05A5E7BBA5@MX14A.corp.emc.com>
In-Reply-To: <7C4DFCE962635144B8FAE8CA11D0BF1E05A5E7BBA5@MX14A.corp.emc.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: dc@ietf.org
Subject: Re: [dc] [armd] IP over IP solution for data center interconnect
Precedence: list

Just saw this:

> If the VM can be moved, then all you need to do is install a temporary
> redirect of packets to the new location. Each host will refresh the MAC
> after 15-30 seconds. If the packets are redirected from old to new
> location for these 30 seconds, the redirect can be aged automatically.
> This happens all the time in mobile networks in what is called a "fast
> handoff" where you redirect the packets until handoff is completed.

"all you need to do" should incur a $5 fine for the network version of
"a simple matter of programming" :-).

I see a bunch of problems here:

Mobile networks aren't running VMs.  ARP caches matter - a little bit of
Googling turns up multiple minute timeouts for some older OS versions and 4
hours for some routers.

In addition, some VMs need static MACs (e.g., software license key or
encryption key may be bound to the MAC).  There will be other reasons why
a VM's MAC cannot be changed when it is live migrated.

This also doesn't cover physical host usage of the burnt-in hardware MAC
on the NIC.  I assume that those need to continue to work as-is.  For
46-bits, I'm guessing that the Hierarchical MAC approach takes all the
locally administered MACs.  If any of those MACs are already in use,
that'll be a problem.

It also looks like it's seriously disruptive to existing L2 network managem=
ent.

I see no mention of TRILL, SPB or PBB in the draft that advocates Hierarchi=
cal
MACs - those are three existing approaches to this problem that can aggrega=
te
without forcing MAC changes on live VM migration.  At the very least, a sol=
id
comparison to existing technology is in order.

Finally, this sort of MAC hierarchy work probably belongs in IEEE, not IETF=
.

Thanks,
--David


> -----Original Message-----
> From: dc-bounces@ietf.org [mailto:dc-bounces@ietf.org] On Behalf Of david=
.black@emc.com
> Sent: Tuesday, January 03, 2012 11:42 AM
> To: adalela@cisco.com
> Cc: dc@ietf.org
> Subject: Re: [dc] [armd] IP over IP solution for data center interconnect
>=20
> Ashish,
>=20
> > > [AD] The higher bits identify a switch - it's a switch-id.
>=20
> That breaks VM migration across switches by forcing a MAC change.
>=20
> Thanks,
> --David
>=20
> > -----Original Message-----
> > From: dc-bounces@ietf.org [mailto:dc-bounces@ietf.org] On Behalf Of Ash=
ish Dalela (adalela)
> > Sent: Tuesday, January 03, 2012 11:15 AM
> > To: robert@raszuk.net
> > Cc: Pedro Marques; dc@ietf.org
> > Subject: Re: [dc] [armd] IP over IP solution for data center interconne=
ct
> >
> > Robert,
> >
> > Please see inline.
> >
> > -----Original Message-----
> > From: Robert Raszuk [mailto:robert@raszuk.net]
> > Sent: Tuesday, January 03, 2012 8:24 PM
> > To: Ashish Dalela (adalela)
> > Cc: Pedro Marques; dc@ietf.org
> > Subject: Re: [dc] [armd] IP over IP solution for data center
> > interconnect
> >
> > Ashish,
> >
> > OK let's just discuss what is in your draft on Hierarchical Addressing.
> >
> > 1. You have 48 bits 32 go for host remaining 16 goes for switches. How
> > do you aggregate at the TOR or AGGR switch boundary ? Are you assuming
> > single HOST - SWITCH with max 65K flat macs ?
> >
> > [AD] The higher bits identify a switch - it's a switch-id. The hosts ar=
e
> > dynamically assigned a host-id under that switch. Let's assume 23 bits
> > are for switch-id and 23 bits for host-id. To forward a packet to the
> > host, you only have to look at the first 23 bits. That's a MAC prefix t=
o
> > route against.
> >
> > [AD] You can have 2^23 switches in a network and 2^23 hosts under each
> > switch.
> >
> > 2. Can you deploy this on existing VMs and existing switches ?
> >
> > [AD] What do you mean by this? Any VM can be configured with any MAC.
> > Any physical host can be configured with any MAC on any logical
> > interface. Configuration standpoint this is possible. Forwarding
> > standpoint, that's another question.
> >
> > 3. What new protocol you envision to use to distribute those new MACs ?
> >
> > [AD] IS-IS extensions. It can be TRILL extensions.
> >
> > 4. What is the advantage of using this vs ILNP if we assume that hosts
> > should be modified ?
> >
> > [AD] I'm not familiar with the ILNP work, but I'm assuming you are
> > talking about Loc-Id separation. If not, correct me. If yes, each Loc-I=
d
> > binding can be a host route, with mobility. These host-routes are a
> > scaling problem. Traditional IP packet have IP as ID and MAC as LOC. We
> > are just extending this LOC to make it actually location aware rather
> > than a flat address which is fixed regardless of where the location is.
> >
> > 5. The proposal does not support aggregation .. even the draft says it
> > :)
> >
> > "The total number of hardware entries anywhere in the network equals th=
e
> >
> > total number of switches and remains agnostic of VM mobility."
> >
> > [AD] For 1 million VM, and 25 VM per host, you need 40,000 hosts. With
> > 48 port access switches, you need 833 switches. That's the routing tabl=
e
> > size for any switch in the datacenter - core, aggregation, access.
> > Contrast this with host-routes, if each VM talks to 100 VMs, then each
> > access switch needs 48 * 25 * 100 =3D 120,000 host routes. Just because
> > the network prefix is 23 bits does not mean we have to store 10^23
> > prefixes. We have to store only as many switches as there are in the
> > network. Ratio between VM : switch is 1000 : 1 (today, assuming 48 port
> > access and 20 VM per port). That means instead of storing host-routes
> > which will grow proportional to VM growth, we store switch-id, which
> > will grow at 1000 times slower rate. As VM density increases, this
> > growth rate is further slowed down. There are other techniques to
> > further reduce the rate of growth. But in any case, 1000 times slower i=
s
> > a lot slow.
> >
> > So if I have 100K switches I can not do any aggregation and need to
> > "route" 100K MAC addresses.
> >
> > [AD] I don't know how you came to that conclusion. Think of HMAC as an
> > IP address. Instead of 32 bits it is 46 bits. You route by prefixes in
> > L3, and you are routing by the same prefixes here. Just as you aggregat=
e
> > IP, same way you aggregate MAC. It's not different.
> >
> > 6. Who provides me the mapping between switch mac and host/vm mac behin=
d
> >
> > such switch ? Do switches proxy arp globally within your domain ?
> >
> > [AD] Variation of the same question. Above should answer it.
> >
> > Thx,
> > R.
> >
> >
> > > Robert,
> > >
> > >>> So you are advocating solution which is based on encapsulation -
> > that
> > > is fine.
> > >
> > > No, I'm not. Did you read the draft I had mentioned?
> > > Hierarchical MAC is not encapsulation. It is one 48 bit address.
> > >
> > >>> However how could you ever arrive at the conclusion that HMACs woul=
d
> > >>> scale better then "anything we know". Well I don't know about you,
> > > but I
> > >>> know that the key to scaling is ability to aggregate. And it is not
> > > that
> > >>> huge mystery that MACs aggregate rather poorly while there are quit=
e
> > >>> well deployed protocols (be it IPv4 or IPv6) which aggregate
> > natively
> > >
> > > You are hitting the issue on the nail. So, read the draft I mentioned=
.
> > > Hierarchical MAC is higher bits "network prefix" and lower bits "host
> > > id".
> > > That's summarizable like IP address and aggregated.
> > > It has 46 bits to modify so larger than IPv4 internet.
> > >
> > > I won't comment on the rest, because you have made an assumption abou=
t
> > > encapsulation.
> > >
> > > I refer to this -
> > > http://tools.ietf.org/html/draft-dalela-dc-approaches-00.
> > >
> > > Thanks, Ashish
> > >
> > >
> > > -----Original Message-----
> > > From: Robert Raszuk [mailto:robert@raszuk.net]
> > > Sent: Tuesday, January 03, 2012 7:05 PM
> > > To: Ashish Dalela (adalela)
> > > Cc: Pedro Marques; dc@ietf.org
> > > Subject: Re: [dc] [armd] IP over IP solution for data center
> > > interconnect
> > >
> > > Ashish,
> > >
> > >> The issues of scale you mentioned don't exist in Hierarchical MACs,
> > >> which scale better than anything we know of.
> > >
> > > So you are advocating solution which is based on encapsulation - that
> > is
> > > fine.
> > >
> > > However how could you ever arrive at the conclusion that HMACs would
> > > scale better then "anything we know". Well I don't know about you, bu=
t
> > I
> > > know that the key to scaling is ability to aggregate. And it is not
> > that
> > >
> > > huge mystery that MACs aggregate rather poorly while there are quite
> > > well deployed protocols (be it IPv4 or IPv6) which aggregate natively=
.
> > >
> > > For inter-dc this is IMHO a must. A must even if you build it using
> > > traditional routers or OF enabled switches - does not matter.
> > >
> > >> I don't want to split the requirements into multiple use-cases
> > >> because then this DC group will be many groups - one doing L2 and
> > >> another doing L3. That I think you will agree is not optimal for
> > >> anyone
> > >
> > > Why MAC-in-IP does not solve it for everyone ? And there are deployed
> > > solutions already ..
> > >
> > > IMHO what this group should accomplish is not to try to reinvent the
> > > world, but perhaps as example discuss where is the right boundary of
> > > encapsulation, how should we communicate between network and hosts,
> > what
> > >
> > > kind of DC instrumentation should be IETF blessed for easy integratio=
n
> > > (ie min subset of functionality it should possess etc .... )
> > >
> > > R.
> > >
> > > _______________________________________________
> > > dc mailing list
> > > dc@ietf.org
> > > https://www.ietf.org/mailman/listinfo/dc
> > >
> > >
> >
> > _______________________________________________
> > dc mailing list
> > dc@ietf.org
> > https://www.ietf.org/mailman/listinfo/dc
>=20
> _______________________________________________
> dc mailing list
> dc@ietf.org
> https://www.ietf.org/mailman/listinfo/dc