Re: [v6ops] Comments on <draft-horley-v6ops-lab>

[Kevin Myers <[e-mail]>]

It seems odd to me that the “size” of the prefix is even an concern in an IPv6 discussion for a /7 that’s been mothballed for 15 years and should already be filtered. 

Speaking as someone who works for operators around the world, there is a very real need for this. We’ve done work for hundreds of ASNs and the larger they are, the more time we spend in network modeling. ULA precedence creates too many problems when testing end systems in lab networks intended to replicate a prod ASN.

When building, operating and modeling ISPs, we have to bolt on a slew of compute systems with various OSes like billing, QoE, traffic engineering controllers, monitoring, EPCs, DNS, DHCP, etc, etc. Modifying the precedence for all of these systems in the lab to verify prod operation is an unreasonable ask when we have plenty of space to properly represent networks and systems in GUA. It’s hard enough in general to get support systems to properly use IPv6 in the control/management plane without fighting an additional battle over ULA precedence. Also, some vendors in the ISP space aren’t going to give you access to the underlying OS so it’s not practical to expect that every OS can even be modified – nor would we want to. This is *especially* true as we accelerate rollout of private LTE/5G and CBRS systems that come with a boatload of control systems.

One of the biggest hurdles in using the other alternative suggested - real RIR space - is safety:

 I am currently working on a design for a service provider ASN that’s acquired 20+ ISPs – all of which have their own IPv6 allocations/ASNs and will eventually merge into a single ASN. So that’s 20 /32s (or larger) that we currently don’t have a great way to model and ULA is useless because the ISP support systems mentioned above will treat ULA inconsistently.

 We are building a Proof of Concept for the new ASN which will include a new route reflector hierarchy, backbone and support systems…at some point we’ll attach the POC to the prod networks and test the new routing design before bringing each ASN fully into the new collapsed design. Having a GUA lab space that can be safely tested with prod to facilitate large merger and acquisitions like this is key to facilitating IPv6 adoption.

It's also worth noting that as we move network operations over to IaaS and NetDevOps methodologies, we are also automating lab development and deployment. It's not a big deal anymore to spin up thousands of routers and model large networks because we can do it programmatically very quickly. As such, we have a big gaping hole in a proper IPv6 GUA lab space to design the global networks of the future.

We don’t need impediments to IPv6 adoption, there are plenty of those already. We need to lower the barrier to entry and as much as we’d like to pretend otherwise, ULA and/or RIR allocations do not best solve every use case for network modeling/development to support real world operations.

Kevin Myers

-----Original Message-----
From: Nick Buraglio <[e-mail]> 
Sent: Tuesday, August 3, 2021 8:52 AM
To: Mark Smith <[e-mail]>
Cc: Brian E Carpenter <[e-mail]>; Ca By <[e-mail]>; Bob Hinden <[e-mail]>; IPv6 Operations <[e-mail]>; [e-mail]
Subject: Re: [v6ops] Comments on <draft-horley-v6ops-lab>

There are a lot of comments to address so we have decided to compile the responses into one group reply, in order.


>I've been doing trainings and labs about IPv6 since 2001 and already trained over 75.000 >engineers, probably a bit more. I don't recall any of the participants on the trainings or other >consultancy and deployment activities to have been confused with just the 2001:db8::/32 >prefix, even if the real situation for their own case was a bigger allocation. They were smart >to figure out the equivalence and in fact, forcing them to use that helped in better >understanding how IPv6 addressing works.


>I've trained and consulted people designing networks that required from the RIR allocations >close to /24. Always have been able to do any kind of address planning and lab exercises >using what we have standardized >up to now.


I, too, have been training and doing IPv6 since around 2002. Each of the co-authors on these drafts have also been training and providing
IPv6 consulting for 15+ years. It is interesting you say you can do an address allocation plan for a /24 (assuming you are doing this prior to getting your actual customer allocation) with a /32 of documentation address space. Either by your own admission you are ignoring RFC standards and using some other IPv6 address space not intended for documentation or lab purposes or something else is going on that you are not sharing. We will take the comment under advisement and state that squatting or using someone else’s allocation is a reasonable approach to the problem as opposed to allocating more space (that was already marked as unusable).

 It is true that a /24 is not as common as a /32, however, it is not uncommon for one to receive a /29 or larger if the organization is large. There are a lot of these, it's easy to go and find one. Because of this scale for address planning and appropriate testing, there is absolutely a need for adequate documentation and lab space. What happens now is that the lab spaces often become a Frankenstein's monster of  blocks that don’t really reflect a good design model, or are squatted on space. Neither are optimal, both are problematic.

That said, because those are now far more commonplace, the need to be able to both lab and document at that scale absolutely exists. Some of us in that world see it fairly frequently, thus these drafts address operational and logistical problems that we see in our daily operations.


>I'm not saying I can't be convinced about an additional documentation prefix or something >similar, but not at the scale depicted in those documents.


We are now conflating both drafts, but since we've gone down this path, I will address it. Documentation needs to be able to scale to the size and use cases of a  production network. If it cannot properly represent reality, then the documentation is already unable to meet its core functional requirement.. Since IPv6 is a critical part of The Internet (and is becoming more important every day) the need to be able to trust that documentation can accurately reflect what the practitioners are deploying is critical. Anything else is kicking the proverbial can down the road to be dealt with later, thus further impeding the real-world deployment of IPv6. Expanding the documentation prefix options will help make this easier for many (if not all) organizations. Leaving the documentation prefix as a single
/32 has already proven to be inadequate for many organizations and hampers their ability to collaborate and work on shared IPv6 problems as they adopt IPv6.


>Note that I'm for doing a good/balanced usage of IPv6 addressing space, and that includes >that every subscriber, even residential should get a /48 by default. However, I'm also for a >good balancing of real needs. So not >being ultra-conservationist, but avoiding unnecessary >waste.


We are also cognizant of the need to be good stewards of the community's resources.

Again, we have purposely chosen recycled space that has been officially deprecated for this exact reason. The space suggested in the documentation draft is functionally unused and retired space.
fec0::/10 is deprecated in RFC 3879 (September 2004) and 3ffe::/16 is deprecated in RFC 5156 (April 2008). The lab prefix of 200::/7 is deprecated in RFC 4048 (December 2004). We believe all of these prefixes have been idle and unused for long enough that their intended reuse will have no impact on any organization operating IPv6 today. We are using the prefix sizes previously allocated so there is, quite literally, NO waste, these prefixes would have most likely gone unused for the duration of IPv6’s lifetime given now much unallocated space still remains. The customers and operators we are working with have defined a real need for this. If there is ever a need to reclaim this space, then I posit that reclaiming it from lab and documentation use will be no more difficult than it already is today.


> El 30/7/21 5:12, "v6ops en nombre de Bob Hinden" 
> <[e-mail] en nombre

> de [e-mail]> escribió:


   >  Hi,


 >    I commented at todays V6OPS meeting on <draft-horley-v6ops-lab-01>.  The

> presentation was:


>     
> https://datatracker.ietf.org/meeting/111/materials/slides-111-v6ops-ip
> v6-lab-prefix-draft-00


>     I have now read the draft and the email thread.   My comments are not significantly

> different from what I read in email, but I will put them in my own words by topic.


>     Size of Allocation:


>     Reserving a /7 (specifically 0200::/7) is very very large amount 
> of address to be

> dedicated to “Lab Use”.


>     A /7 is 2^^121, or 2.6584559915698×10^36 addresses.  In words:


>     Eighteen quadrillion fourteen trillion three hundred ninety-eight 
> billion five hundred nine

>  million four hundred eighty-one thousand nine hundred eighty-four addresses.


>     With some sarcasm, how big is this lab going to be?


The labs can scale to the size of a very, very large provider, cloud service, or CDN with their diverse upstream connectivity and adjacent summarized networks, so we’re talking very large sets of large prefix blocks, not lan segments or host addresses.


>  This would allow for 2^57 /64 prefixes, that is 144,115,188,075,855,872 prefixes.   Is your

>  lab going to have that many prefixes?


No, it would not see that many disaggregated prefixes, nor is that in the scope of what we’re trying to do in labs such as this. It’s not about the prefixes, it’s about the summaries. When simulating a series of global networks and their adjacent peers, we start to talk about /24, /29, etc.. And again, this is why we scavenged a deprecated block. A block that, by pure coincidence is already being used for experimental (some might even say lab) purposes by those in the community already, as identified in an email reply back from some commenting on the documentation draft submission.


>     This proposal seems like a waste of address space to me.   The main purpose of IPv6

> was to create a larger Internet, allocating this much space for “lab 
> use” seems very

>  excessive.   We won’t end up with much bigger Internet, if we aren’t careful with how we

>  allocate the address space.


 >    I note for comparison that the original intent for this prefix
was to accommodate the whole > ISO CLNP Internet.


Correct. We purposely chose that space as to recycle something that had been deprecated and laid officially unused for a very long time.
An added bonus was that it is visually similar to the current GUA space. Is there a more functional role you would like to use 200::/7 for? Given that it is already currently being used for this purpose by others in the community, it would make sense to simply designate it and avoid future problems. It is clear that asking organizations who have already deployed it for this purpose NOT to use the address space is more problematic than simply allocating it for this purpose, regardless of if it feels like “a waste of address space.”


>     Global Filtering:


  >  The draft says that this prefix should be added the list of non-routable prefixes.  I note that > this means that every ISP in the
world would need to do this.    That’s a big ask and may not >
actually happen operationally.


In the last 15 years I personally have not seen a network that handles this kind of filtering in a manual, hand curated manner (and yes, I have seen a fair amount of networks in that time). There exists a myriad of options for making this simple based on a number of standard technologies, the most common of which is a BGP feed. Additionally, the filters in question are not something that should be “set and forget”. They should be curated and tended to just as any other type of global filtering is. I see this as a normal piece of maintenance - cost of running in the global DFZ which absolutely should be accounted for anyway / already. We can also submit a notice to Team Cymru to quickly update their larger bogon and blocking list which is used by a large number of organizations. Once their list is updated, that will account for a significant number of networks that will start filtering these ranges.  There are a great deal of resources out there to address this exact situation. If folks would like to know more about them, please contact me and I will be happy to provide more information.

Since we are completely open to being out in left field here, we leave it to other operators to chime in as to how their bogon-style filtering is done, but in our collective experience this is a normal, ongoing maintenance task. Additionally, if I am mistaken or misinformed and it is both atypical AND as operators we are reluctant to update BGP filters and/or ACLs for things that change as much as this, then I personally believe we’re heading in the wrong direction with the DFZ and best practices filtering, but I digress, as that is a very different problem.


>     Unintended Consequences:


   >  As noted on the email thread, this is going to have the unintended consequences of

 > recreating a private IPv6 address space.  The IETF has gone to a
lot of trouble to not have > private address in IPv6.   The original
design has Site Local addresses.  These were

> deprecated, for very good reason described in RFC 3879.


Agreed, and that is a valid point. However, due to reasons stated above and experience in lab deployment operations, this does not provide a realistic operational experience. It clouds the operator to the realistic functionality of deploying IPv6. As a community we need to be able to often and openly re-evaluate past decisions or we risk becoming stagnant, which in turn hinders deployment (and thereby
progress) by ignoring real operational impediments to deployment. We are not advocating for change for the sake of change, only the opportunity to change where there are actual benefits to those of us actively pushing out and evangelizing IPv6 daily, and to address needs and requirements of those working with these technologies. In addition, it should be noted that the IETF effectively built private addresses in IPv6 with the introduction of ULA. There is not a single networking professional I teach or engage with around IPv6 for the first time that doesn’t immediately coorolate ULA with RFC 1918 IPv4 address space. If others see a lab prefix as private IPv6 address space that is fine, because they do that already for ULA, that behavior cannot be changed, regardless of what the intent was/is for ULA.


>     The draft call out that the allocated prefix should be declared 
> “non-routeable” and be

> added to packet filters.   Exactly what would be needed to recreate a new form of Site-Local > addresses.


   >  Unique Local Addresses (ULA):


>     The draft describes why it can’t use ULA addresses.   The ULA prefix also happens > to be a /7 and can support the same number of prefixes as what is proposed in the draft.


>     Regarding special handling of ULA in source address selection.   That is default behavior > that can be changed. It’s also not clear that the old NSAP prefix (0200::/7) will not have a > > similar or different set of issues, given it is currently reserved and not in the address blocks > that are currently allocated for global unicast.


While admittedly not exhaustive, my testing so far is that 0200::/7 works as GUA, whereas ULA does not (again, by design). Changing this default behavior in host operating systems is unrealistic, even at a small scale. While it may be possible to make host tweaks in mainstream operating systems, this devolves into “not possible” very quickly. Even with the linux distributions, there is no consistency in how the preference is set, nevermind things like embedded systems, which already lag behind in IPv6 support. These devices and this amount of manual host OS work - even with automation - are the main impediments we see as architects, engineers, operations and deployment practitioners dealing with on a regular basis in lab environments.
Even given the further comments down the thread regarding
getaddrinfo() and address selection, it does not change the fact that ULA operates differently than GUA. Since we cannot realistically expect an operator to be able to adjust that preference value in every single device in a lab, we are left with squatting on GUA space, thus adding unnecessary risk and uncertainty to a given lab environment. In the following example, we can further demonstrate the behavior of having GUA, ULA and IPv4 in a routed lab:


Use of DNS for a dual stacked device


(~) imac5k $ host !$

host gw-starlink

gw-lab-transit-north.buragl.io has address 10.255.255.3

gw-lab-transit-north.buragl.io has IPv6 address fd68:1e02:dc1a:ffff::3


The connecting host is dual stacked, it also has a fully routed ULA and the standard array of GUA addresses, in addition to an RFC1918
IPv4 address:


(~) imac5k $ ifconfig en0

.....<snip>
inet 10.209.4.9 netmask 0xffffff80 broadcast 10.209.4.127

inet6 fe80::18fe:940b:576e:9c0c%en0 prefixlen 64 secured scopeid 0x4

inet6 2607:xxx:xxx:xxx:xxx:xxx:xxx:xxx prefixlen 64 autoconf secured

inet6 2607:xxx:xxx:xxx:xxx:xxx:xxx:xxx prefixlen 64 autoconf temporary

inet6 fd68:1e02:dc1a:4:80b:a1e9:555c:2459 prefixlen 64 autoconf secured

nd6 options=201<PERFORMNUD,DAD>

media: autoselect (1000baseT <full-duplex,flow-control>)

status: active


Even with the presence of ULA AND GUA, fully routed through the network, it defaults to IPv4.


[buraglio@gw-lab-transit-north] > /user active print

Flags: R - radius, M - by-romon

 #    WHEN                 NAME                              ADDRESS
                                                         VIA

 0    aug/02/2021 11:36:46 buraglio
10.209.4.9
ssh


Please excuse the formatting. However, as is evident even with a ULA address AND a GUA address on the client host, since the DNS for my router loopback is ULA, we default to IPv4. This is pretty easily reproducible across platforms. This test happened to be performed using a Mac running the latest patches as the client and a RouterOS device.



>     I strongly that ways can be found to use ULAs to meet the requirements of the “lab use”  > proposal.   For example, if you need n prefixes, run the algorithm n times and get that

> number of random prefixes.   Each prefix has have 16-bits for a Subnet ID, that can used to > prefix aggregation.   None of this requires any change of standards, allocations, or

> additional global filtering.


We see operational issues with using ULA due to RFC6724. Changing the default address selection across any scaled number of devices - especially embedded equipment - including network hardware - is functionally impossible. Changing RFC6724, as stated, is undesirable and likely would take too long for host OS manufactures to change, even if a change was decided. The goal is to have a usable lab prefix space and not have to change RFC 6724 as the impacts of that are unknown. There are still a large number of devices and host OSs out there using RFC 3484 settings for example. We are asking those adopting IPv6 to have to classify all their devices as RFC 3484, 6724, whatever new RFC to update 6724 theoretically being proposed, and those devices that completely ignore 3484, 6724 and do something different. Operationally, this is not desirable and introduces far too much complexity when testing and validating.



>     I think that if you want something beyond what ULAs can 
> accommodate (and can

> demonstrate a need), you will need to describe that in a lot more 
> detail than what is in the

>  current version of the draft.  I don’t see anything in the draft that 
> justifies an address

>  allocation of this size for “lab use”.


The size is opportunistic due to reusing a deprecated prefix. As stated above, it is a large enough space to accommodate the global scale, but also a recycled address block to conserve and not be wasteful. With IPv6 we espouse that we should not concern ourselves with the sizes of address blocks in the same way that we do IPv4, and while that is noble, we know that there are limits. We also need to remain good stewards of the community resources while allowing for the existing environments to flourish.


At the end of the day, what we’re hoping to accomplish is to make testing, prototyping, sharing, and eventually deployment easier, and both of these drafts address very real hurdles we see quite frequently. We are sharing these needs as operators, designers, architects and engineers of IPv6 networks. We wouldn’t ask if there wasn’t a need and we are not trying to be egregious in our request by being mindful of the prefixes we are proposing.

Thank you for consideration,

nb
---
Nick Buraglio
Planning and Architecture
Energy Sciences Network
[phone]

On Fri, Jul 30, 2021 at 8:20 PM Mark Smith <[e-mail]> wrote:
>
> On Sat, 31 Jul 2021 at 07:42, Brian E Carpenter 
> <[e-mail]> wrote:
> >
> > On 31-Jul-21 05:20, Ca By wrote:
> > >
> > >     Global Filtering:
> > >
> > >     The draft says that this prefix should be added the list of non-routable prefixes.  I note that this means that every ISP in the world would need to do this.    That’s a big ask and may not actually happen operationally.
> > >
> > >
> > > I agree with Bob. Don’t do this to us operators.
> > >
> > > Broadly, i oppose the i-d. It is not needed. Any lab can use gua 
> > > or ula
> > or bogon
> >
> > I agree. However, I did see one comment earlier: that the RFC6724 precedence table needs to be tweaked in all hosts to avoid special handling for fc00::/7. Is there a good way to do that remotely?
> >
>
> I've been wondering if that is really the case, or there is something 
> else going on in the example that Nick posted to the list.
>
> The only way for IPv4 to be preferred over ULA per the precedence 
> table is if IPv4 addresses are represented as IPv4-mapped addresses 
> (per RFC6724, 3.2).
>
> The example we've seen on the list was with SSH under Ubunto, so using 
> glibc getaddrinfo().
>
> AI_V4MAPPED | AI_ADDRCONFIG are the default flags for getaddrinfo() if 
> NULL is specified for the 'hints' structure for the getaddrinfo() 
> call.
>
> However, the call to getaddrinfo() in openssh-portable in the current 
> github version (ssh.c, resolve_host()) always supplies a 'hints'
> structure, and doesn't ever set AI_V4MAPPED.
>
> Here's a getaddrinfo() test using the same flags that SSH uses by default:
>
>     memset(&hints, 0, sizeof(hints));
>
>     hints.ai_family = AF_UNSPEC;        /* Allow IPv4 or IPv6 */
>     hints.ai_socktype = SOCK_STREAM;    /* Stream socket */
>
>     s = getaddrinfo(argv[1], NULL, &hints, &result);
>
> looking up a domain name with only IPv4 and IPv6 ULA DNS records. The 
> output order shown is the order getaddrinfo() supplied:
>
> [mark@opy getaddrinfo]$ ./gai v6opstest.nosense.org
>
> getaddrinfo(v6opstest.nosense.org)
>
>     AF_INET6, fd68:1e02:dc1a:ffff::3, sin6_scope_id = 0
>     AF_INET, 10.255.255.3
>
> [mark@opy getaddrinfo]$
>
> So the ULA address should be preferred as a DA over IPv4.
>
> I'm wondering if perhaps the preference of IPv4 over IPv6 ULA was 
> instead caused by source address selection. Perhaps Nick's test host 
> only had IPv4 and IPv6 GUA addresses, so IPv4 became the SA choice 
> when there was only IPv6 ULA and IPv4 DAs. OTOH, if the DA choices 
> were IPv4 and IPv6 GUA, then IPv6 GUA was the chosen source address.
>
> So perhaps source address selection made it appear that IPv4 was 
> preferred over ULA for destination address selection.
>
> Regards,
> Mark.
>
>
>
>
>
>
>
>
>
> > There's also an argument in the draft that RFC4193 specifies pseudo-random bits in the prefix. In the lab context, that's a requirement than can be (and often is) ignored.
> >
> >     Brian
> >
> > _______________________________________________
> > v6ops mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/v6ops