Re: [v6ops] I-D Action: draft-fbnvv-v6ops-site-multihoming-00.txt

[Brian E Carpenter <[e-mail]>]

Just a couple of extra comments where I was unclear in my first message:
On 05-Mar-23 01:55, Klaus Frank wrote:
> Hi
> 
> On 04.03.2023 04:09, Brian E Carpenter wrote:
>> Hi,
>>
>> Some comments (not really a full review):
>>
>> Firstly, thanks for starting this work, it is badly needed.
>>
>> Second, a bit off topic, but it was recently rediscovered
>> where I live (New Zealand) that any amount of MHMP won't help
>> you if your region is supplied by a single fibre trunk shared
>> by all providers, and a landslide removes that trunk. It might
>> be worth reminding readers of that. We really need MHMPMD
>> (MD = multiple ducts).
> You could still get Starlink as a redundant uplink for example.
>>
>> One of the four solutions reads:
>>
>> "- Dynamic PA addresses distribution and withdrawal from carriers"
>>
>> I think that is really hiding a more traditional solution,
>> something like:
>>
>> - Static PA addresses from multiple carriers, chosen dynamically
>>    by hosts.
> 
> No, thous two aren't the same. The first one is where the ISP also
> occasionally assigns you a different one. So your PA prefix changes
> dynamically. Or at least when your router reconnects. So it isn't a
> static PA allocation at all.

I expressed myself badly. I think you should in fact list 5 solutions,
with "Static PA addresses" before "Dynamic PA addresses".

Of course, even static prefixes are not static for ever, but for large
enterprise networks a change of PA prefix would be a planned event
(RFC4192) and the MHMP problem is simpler.

I agree of course that the dynamic case must also be solved.

> 
>>
>> Dynamic prefix assignment and withdrawal is a *new* model.
>> Whether it can be made to work is an open question.
> We need it to work nonetheless, as providers often tier their services
> to have dynamically allocating and therefore changing prefixes on the
> cheaper contracts and often only offer static ones either at a higher
> price at a very high price together with SD-WAN offerings...
>>
>> I'm ranting about this because whatever solution we propose must
>> satisfy the needs of end hosts at least as well as it satisfies the
>> needs of enterprise network operators and carriers. And that aspect
>> isn't clear to me in the list that starts "4. Solution requirements".
>> (I'd personally prefer a solution that requires no new action
>> whatever by the carriers.)
> 
> That's why we listed multiple. Our assumption was that Enterprises will
> want to use their own PI prefix and either use a tunnel to a centralized
> point (similar to LISP) or have SD-WAN contracts with the ISP and be
> able to speak BGP with them to route the PI directly.
> 
> Where as smaller and home entities may either use a service provider
> (LISP), dual assignment of all the prefixes if it's a flat hierarchy
> "plug both ISP routers into the same local network", or use a
> (cheap/simple) load balance in front of the CPEs and do NAT/NPT on it.
> 
>>
>> One of the requirements:
>>
>> "4. Sub-second convergence for the prefix deprication..."
>>
>> comes out of nowhere. Won't this vary widely? In a small office
>> network, why not 5 seconds or 30 seconds? In a stock trading
>> data centre, wouldn't they want sub-millisecond convergence?
> The intent behind this was to stay within the usual timeout windows of
> higher protocols to allow for a re-transmission without a connection
> error being user-facing.
>> Also cover the case of redundant topologies (>1 router per subnet).
> 
> Outside of the scope of this RFC and where relevant already covered. If
> you have two redundant routers on the subnet, then they'll route it
> towards the CPEs at some point. And at that point it is the same as if
> it was a client that sent the package. Also we mentioned the concerns
> about complex networks and what approach would allow for static
> subnetting and which ones don't.
> 
> The main issue this whole proposal is that ISPs don't offer BGP to
> customers at an acceptable price point and that if everyone had their
> own PI and would use BGP to the ISP that it would bloat the global
> routing table. Another ideal world solution would be that IP addresses
> are allocated by geographical area (and thereby allow route aggregation
> using it) and not legal entity and a RPKI like approach for the ISPs to
> assign IPs and allow customers to use any "RPKI-valid" source prefix
> through their network and route it into the internet even if it was
> assigned by another ISP.
> 
> But that's all make believe and we have to deal with the cards we were
> given, that's also why we explicitly mentioned issues with cellular and
> NAT66...
> 
>> "Privacy is furthermore assumed to be protected by [Temporary
>> addressing] and is also kept outside of the carrier resiliency
>> design consideration."
>>
>> I think that needs rewording. Some enterprises forbid temporary
>> addresses, so the assumption is sometimes false. I think it would
>> be safer to say that address-based privacy considerations are
>> not affected by carrier resiliency in itself. (They might be
>> affected by the choice to use or not use NAT66 or NPTv6, but that
>> is a separate discussion.)
>>
>> "5.1. PI-based
>> ...
>> Advantages:"
>>
>> Isn't the #1 advantage: It works today! ?
> 
> Because it doesn't like it should. It only works in principal but has
> huge issues in practice for most people...
> 
>> "Disadvantages:
>> ...
>> - Bloats the Internet routing table."
>>
>> How about
>>
>> - Bloats the Internet routing table with a potential 10 million new
>> routes. ?
>>
>> I don't think we should make this solution sound satisfactory.
> 
> Well, that still wouldn't be a concern of the entity that wants to have
> redundancy. And ultimately we don't really have any solution that
> doesn't have issues right now.
> 
> The only real way to change this would be to completely rethink the
> address assignment (see make believe above) and redo all of it, so it's
> basically unavoidable/unsolvable at this point...
> 
>>
>> "5.2. PA-based solution
>> ...
>> It is mandatory to implement requirement L-13 of [HNCP] section 11
>> to deprecate delegated prefixes..."
>>
>> That reads as if you are proposing HNCP for everyone, which I am sure
>> is not your intention. Can you rephrase it to sound like a property
>> of the solution?
> 
> It already is, as it is listed within 5.2 and all sub sections of 5 are
> different individual solutions.

Again I expressed myself badly. I don't disagree technically, but the
current text can mislead the reader to believe that you are suggesting
HNCP as the solution. I'm suggesting that your text should be something
like:

"As soon as a CE router becomes aware that the path to the carrier that
delegated a prefix is down, it is necessary that all more specific
prefixes under that prefix are deprecated (preferred lifetime set to
zero) in all subnets. This is specified as requirement L-13 of [HNCP]
but is not currently available in other routing mechanisms."

This leads directly to a requirement statement for the general case,
where HNCP is not relevant.

     Brian

> And it is in fact mandatory to the solution 5.2. So if you want to use
> the suggested PA-based solution you'll have to use it...
> 
>>
>> "Potentially, the application on the host may use bind() to choose
>> the source address..."
>>
>> Yes, but most applications don't do that, they start with getaddrinfo().
>> So what is the value of this discussion of bind()?
> Because it is an issue with application compatibility and the whole
> point of this proposal is to have a resilient up-link to not have
> applications loose internet link and have a solution that does not break
> applications. So this case sadly has to be considered as it is affecting
> too many applications to just be ignored. But I admit that we'd need
> better statistics for that to not just be based on individual
> experience, but we currently don't have such statistics nor a good way
> to get them...
>>
>> "It may be valuable to implement [Provisioning domains]..."
>>
>> What is the probability of that happening?
>>
>> "Disadvantages:
>> ...
>>     - Carriers may frequently change the prefix (flash renumbering), and
>>       this could disrupt communication"
>>
>> Why is that new? Many IPv6 carriers do that for SOHO users today. I think
>> they don't do it for larger enterprises, though.
> It depends, I know from the top of my head of at least one enterprise
> that had within the IPv4 world it's redundant links on the cheapest
> contract available. So they'd do it there as well... And within the IPv4
> world they also didn't have any real reason to get a more expensive
> offer, as they accomplished the same SLA level by just having multiple
> redundant up-links and saved a non significant amount of money compared
> to using the more expensive contracts that didn't and because they used
> NAT44 the public IP change also didn't affect the internal network at
> all. Even for the VPN connections between the sites they just used
> dyndns and had the CPEs update the IP in DNS...
>>
>> "5.3. ULA with NPT
>> ...
>>     However, [ULA] has
>>     notable limits concerning the number of sites that one prefix may
>>     span, and well documented usability limitations when considering
>>     address selection details across large, diverse network
>>     environments."
>>
>> I'm not sure these limits are "well documented". If they are, please
>> give a reference.
> * RFC6724
> * documentation of the major operating systems (Windows, MacOS, Fedora,
> RedHat, ...)
> * documentation of the major firewall vendors (PaloAlto, Sophos, Cisco, ...)
> * almost all it handbooks that talk about network fundamentals and also
> include IPv6
> * countless blog posts with advice on how to configure IPv6
>> "5.4. ULA with NAT66
>> ...
>> Advantages:
>> ...
>> - NAT may be a normative requirement in itself (this is highly
>>       questionable, but nonetheless brought forward in many
>>       discussions)"
>>
>> Note that PCI-DSS (version 4, March 2022) does not require NAT,
>> although it is mentioned as a solution for IPv4. See
>> https://docs-prv.pcisecuritystandards.org/PCI%20DSS/Standard/PCI-DSS-v4_0.pdf
> That's exactly what we were referring to by "this is highly
> questionable, but nonetheless brought forward in many discussions", even
> here on the IETF lists there were discussions not even two months ago of
> working together with regulatory entities to avoid (and where it exists
> currently) remove NAT66 requirements...
>>
>> " 6. Conclusion"
>>
>> I think in general this section is premature, until the earlier sections
>> have been widely reviewed and updated. So mainly I will not comment, but:
>>
>> "For sites having a complex topology (many links and routers), a PA-
>>     based solution is not an option yet,..."
>>
>> Agreed, but:
>>
>> "... because it would need automatic
>>     PA address distribution over the site and neither [HNCP] nor DHCP-PD
>>     have gained market acceptance for this task."
>> That is needed for flash renumbering, but NOT for a static multi-PA
>> solution. There, the requirement is much simpler: prefix up/down
>> status distribution over the site. That's close to a trivial problem
>> to solve, if we decide to solve it. (For example, it would be easy
>> to define a GRASP [RFC8990] method for it, and I'm sure there are
>> ten other solutions waiting.)
> 
> Right, we kinda neglected the part of having only static PA allocation
> by the ISPs. Almost the whole draft is assuming that the PA prefix can
> and will change without any upfront warning. But even if we for now
> assume the PA allocated by every ISP is static, then we still have the
> source address selection issue. As the ISPs still would filter the
> packages that we push their way that have a Pa allocation of any of the
> other ISPs in the mix... But I agree that in the case of static PA
> allocations a customer could still do a full switch over to a different
> link. It's only a active-passive setup. Still not ideal (active-active
> with the ability for traffic engineering by the site admin), but better
> than nothing for sure.
> 
> Side note, please don't take this too serious right now, it's not fully
> worked out, just something I had to think about reading your reply to
> this: What could help even in the case of multiple dynamic PA
> allocations is if we'd have DHCP-PD solutions that could dynamically
> update their configuration based on what the CPE provides (router
> advertisement or DHCP-PD, also UPNPv6 could help if it was reliable like
> it's IPv4 equivalent). But I don't think any network equipment currently
> supports such a "feature" (but custom scripting on Linux may work 🤔
> right now), but at the end that's just a worse implementation of what we
> have right now. - Or is it? After all a solution that doesn't involve
> the ISP or CPE is exactly what we need to unblock our selves from these
> limitations...
> 
>>
>> Regards
>>     Brian Carpenter
> 
> Thanks for your review of our proposal.
> 
> Sincerely,
> Klaus Frank
> 
>>
>> On 04-Mar-23 03:59, [e-mail] wrote:
>>>
>>> A New Internet-Draft is available from the on-line Internet-Drafts
>>> directories.
>>>
>>>
>>>           Title           : IPv6 Site connection to many Carriers
>>>           Authors         : Nick Buraglio
>>>                             Klaus Frank
>>>                             Paolo Nero
>>>                             Paolo Volpato
>>>                             Eduard Vasilenko
>>>     Filename        : draft-fbnvv-v6ops-site-multihoming-00.txt
>>>     Pages           : 32
>>>     Date            : 2023-03-03
>>>
>>> Abstract:
>>>      Carrier resilience is a typical business requirement. IPv4
>>>      deployments have traditionally solved this challenge through private
>>>      internal site addressing in combination with separate NAT engines
>>>      attached to multiple redundant carriers. IPv6 brings support for
>>>      true end-to-end connectivity on the Internet, and hence NAT is the
>>>      least desirable option in such deployments. Native IPv6 solutions
>>>      for carrier resiliency, however, have drawbacks. This document
>>>      discusses all currently-available options to organize carrier
>>>      resiliency for a site, their strengths and weaknesses, and provides
>>>      a history of past IETF efforts approaching the issue. The views
>>>      presented here are the summary of discussions on the v6ops mailing
>>>      list and are not just the personal opinion of the authors.
>>>
>>>
>>> The IETF datatracker status page for this Internet-Draft is:
>>> https://datatracker.ietf.org/doc/draft-fbnvv-v6ops-site-multihoming/
>>>
>>> There is also an htmlized version available at:
>>> https://datatracker.ietf.org/doc/html/draft-fbnvv-v6ops-site-multihoming-00
>>>
>>>
>>>
>>> Internet-Drafts are also available by rsync at
>>> rsync.ietf.org::internet-drafts
>>>
>>>
>>> _______________________________________________
>>> I-D-Announce mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/i-d-announce
>>> Internet-Draft directories: http://www.ietf.org/shadow.html
>>> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>>
>> _______________________________________________
>> v6ops mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/v6ops
>>
>> _______________________________________________
>> v6ops mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/v6ops