Re: [v6ops] MAC table shortage in IPv6 networks caused by multiple IPv6 prefixes/addresses//FW: New Version Notification for draft-liu-v6ops-running-multiple-prefixes-01.txt

["Liubing (Leo)" <[e-mail]>]

Hi Andrew,

> > [Bing] Would you like to share more details of the network? E.g. it is a
> dual-stack or something else.
> 
> It's dualstack, the vast majority of the clients are on a single wireless
> segment (the total number of wired devices is ~300-400, so I treat them as
> noise.
> 
> The wireless clients are addressed with SLAAC, and are outside of my control
> (conference attendees).
> 
> The segment has no officially designated servers.
> 
> The configuration from the first hop router is below:
> 
> interface Vlan4
>   description !!! WIRELESS CLIENTS !!! CiscoLive2014 !!!
>   .. IPv4 configuration omitted ..
>   ipv6 address FE80::1 link-local
>   ipv6 address 2001:4D38:A:400::1/64
>   ipv6 nd reachable-time 1800000
>   ipv6 nd prefix default 86400 9000 off-link no-autoconfig
>   ipv6 nd prefix 2001:4D38:A:400::/64 604800 86400 off-link
>   ipv6 nd prefix 2001:4D38:A:6800::/64 604800 0 off-link
>   ipv6 nd router-preference High
>   ipv6 nd ra lifetime 9000
>   no ipv6 redirects
>   no ipv6 unreachables
>   ipv6 verify unicast source reachable-via rx
> 
> ipv6 route 2001:4D38:A:400::/64 vlan4
> 
> Note a few bits that are specific for the conference setup, and some are
> specific to the topology:
> 
> 1) maximally high lifetimes (RAs are throttled on their way to the clients
> down to 1 RA in 10 minutes).
> 
> 2) clearing on-link bit: results in clients never trying to resolve the other
> clients' addresses => less memory usage on the portable devices, smaller ND
> caches there.
> 
> 3) default no-autoconfig: to force the necessity to add the prefixes manually,
> to force the operator to always have the "ipv6 nd prefix ..."
> with the correct config.
> 
> 4) two prefixes, one with zero preferred lifetime: there was a second VLAN
> for disaster recovery purposes, where the clients would have ended up if
> there was a catastrophic failure no the main pair of the wireless controllers.
> That VLAN had a mirrored configuration.
> Experimentally, I've found such a configuration to cause a rapid deprecation
> of the "wrong" prefix on a iOS and OSX, resulting in minimal disruption to
> them in case of switchover.
> 
> 5) interface route for the prefix, to send the traffic to it (configuring the
> prefix "off-link" had removed the directly connected route).

[Bing] Andrew, thanks much for sharing the details. 

> > It is quite often that a host would have at least 3 IPv6 addresses: a
> >link-local, a global and a privacy address. If ULAs are enabled,
> >DHCPv6/SLAAC co-existing, there might be more IPv6 addresses.
> 
> Right. I was well within the constraints for the first hop, so opted to not have
> DHCPv6. If I wanted a tighter control, I'd have turned off SLAAC and used
> stateful DHCPv6. Maybe I'll experiment with it this year.
> 
> > So I was just guessing, maybe the 1.7x only represent that part of the
> nodes is enabling IPv6?
> 
> The %% of the nodes running IPv6 was about 80-85%
> (http://2014.ciscolive-ipv6.com/munin/ipv6noc/ipv6noc/neighbors_dualstac
> k_percentage.html
> - the dips are the the night times, a lot of internal apps were IPv4 so the
> neighbor entries went away)
> 
> I calculated this %% by taking the "show arp" and "show ipv6 neigh"
> outputs and correlating the hosts by mac address.
> 
> As I said Apple devices (they are the most aggressive today wrt the privacy
> addresses) are ~50% of the network, so maybe in another setup the figures
> would be different.
> 
> Do you have some numbers you could share ?

[Bing] The problem was encountered by the co-author (Yang Bo) when doing projects in some campus/enterprise networks. I'm sorry we don't have the specific statistics data like above, since we don't own the networks. 

Now there are some enterprise/campus networks under real use or considering using L2 networks. Some are aiming at better user isolation through VLANs (some even consider QinQ mechanism); while some are aiming less configuration/management than the traditional L3 networks. So there would be thousands of hosts aggregated to the core switch (normally there are two core switches stacked together, but only share one cache space). As IPv6 is beginning real use, for example, some of the campus networks are already dual-stack, and the majority of the hosts are Win 7, we once observed in one campus that DHCPv6/SLAAC are both enabled, each Win 7 host had 4 IPv6 addr (SLAAC+DHCPv6+Privacy+link-local)+1 IPv4 addr. 

Current high-end switch can certainly fulfill the thousands of users' IP-MAC entries, however, the forwarding performance would be a significant redundancy, which would cost more budget than expected. This is something that won't happen in IPv4, so I thought maybe this could be considered as an IPv6 networks operational issue, and took the issue into the draft and gain some opinions in the list.

> > Plus, an IPv6-MAC binding would cost more cache space than an IPv4-MAC
> binding (2-4 times due to implementation).
> >
> >> I re-read the original mail starting this thread and the problem
> >> description there to me boils down to two issues:
> >>
> >> 1) one IP node has more IP addresses in version 6 than in version 4.
> >> 2) an IP address takes more lookup space in version 6 than in version 4.
> >>
> >> Do I grok it right ?
> > [Bing] Basically yes. This is one of the problems cause by multiple
> > IPv6 addresses in draft-liu-v6ops-running-multiple-prefixes.
> 
> Ok, then I think that's a tricky one - we can not decrease the number of bits
> in IPv6 address, nor get the end hosts to go back to ARP for address
> resolution :-)
>
> Also, if iOS 8.0 indeed changes the MAC address upon the wireless
> connection, the problem is just about to get much worse and not
> IPv6-specific ;-)
> 
> So, I think there are several means to combat the lookup space churn:
> 
> 1) IPv6-only networks. We can't do that today and IPv4 is relatively not
> *so* much but it's something.
> 
> 2) Table maintenance/cleanup mechanisms. These appear to work today
> reasonably.
> 
> 3) provisioning hardware according to anticipated scale - and either using
> larger router boxes, or splitting large L3 domains into several smaller
> segments, like Mikael mentioned.
> 
> 4) using DHCPv6-only allocation of addresses, with one address per host
> assignment.
> 
> Do you think there is something more that is needed besides the above
> approaches to steer the addresses on the host ?

[Bing] Thanks for the proposed approaches. It seems that's what we can do so far in operation perspective.
As the opinions raised in the mailing list, people tend to consider it as a network design issue rather than operational issue. I will reflect the opinions in the next version of the draft.

Best regards,
Bing

> --a