Re: [v6ops] draft-fbnvv-v6ops-site-multihoming-00.txt

[Klaus Frank <[e-mail]>]

Hi,

On 06.03.2023 16:07, Gert Doering wrote:
> Hi,
>
> On Mon, Mar 06, 2023 at 02:55:41PM +0100, Ivan Pepelnjak wrote:
>>>> Two more disadvantages to the PI-based solution (that I still like most, but as Brian Carpenter said, it might melt down the Internet)
>>>>
>>>> * You might need an ASN
>>>> * Someone will have to deal with RPKI eventually
>>> ... and "if the Internet melts, guess which prefixes people will stop
>>> accepting first".
>> As much as I agree with you, a multinational with sales outlets (or small offices) all around the globe could easily get a /32 (or larger) based on the number of sites it has, carve /48s out of it, and advertise those /48s to DFZ.
>>
>> What are you going to do? Filter out all /48s? Or just the bad /48s? And what is bad? Geoff Huston makes sure (for ages) we know which the bad /24s are every month, but I don???t remember seeing a widespread filtering impact of his reports.
> I'm aware of this, and at some point, these sales outlets will have
> connectivity problems.
>
> In IPv4, we currently do not accept /24 coming from other continent's
> RIRs (saves about 250k FIB slots).  We point a default route at our
> transit ISPs and make it "their problem", but if connectivity is worse
> than it could be, we shrug, and explain that we're not getting any money
> from these entities to make us care.
>
> This is a very classic "tragedy of the commons" problem - every single
> extra slot in the global BGP table is not going to hurt anyone "for real",
> but the sum of it will, at some point, bring meltdown.
>
>
> Why am I spending time on spelling this out?  Because I think that every
> document that spells out "PI is the most technical sound solution"
> (specifically, "BGP with a chunk of independent addresses, whatever
> label the distributing entity puts on them") should also be very clear
> on the consequences, and one of them is "if everyone does this, it will
> break for everyone".

Currently it is a bit less direct. But we're going to add something like 
the "if everyone does it, it'll break for everyone" as Eduard already 
wrote. But even if most people would ignore it, going with the PI 
solution wouldn't be that appealing anyway and they would choose a 
different solution because of multiple other limiting factors (see below 
for my in depth thought process about this).

> I tend to differentiate networks between "there is stuff in these networks
> which can not be reasonably renumbered" - which would be "servers belonging
> to end users", mostly, because coordinating renumbering with "DNS run by
> other people" is truly hard - and "primarily does outbound requests".
>
> The latter can live on dual-GUA, NAT66, ... just fine, provided the IETF
> will eventually fix the source-address selection/failover omissions
> (long-lasting sessions will break, but the Internet moved away from
> long-lasting anything to "all is HTTP chunks", so as long as the *next*
> HTTP request is going to succeed again, who cares about source address?)

Yea, that's basically the main issue and also why we think this proposal 
is required. Also normally I'd agree with you on your differentiation, 
but without the source-address selection issue being addressed these two 
are basically the same as none of them can easily be renumbered as a lot 
of applications even if we deprecate the prefix (because we don't 
de-configure it from the interface) would use it and may even require a 
restart (if poorly written)...

If the source address selection was part of the routing decision this 
whole issue wouldn't exist.

That's why I originally (before we met for this proposal) asked on the 
list about opinions of adding a "has source address filtering" flag to 
router advertisements. So that clients would configure the routes using 
that information (e.g. on Linux: `default via 2001:0DB8::1 src 
2001:0DB8::2` and `default via 2001:0DB8:1::1 src 2001:0DB8:1::2`).

The issue with that solution however is that it would take a lot of time 
until every CPE and OS has that feature and only then everyone could 
rely on it. And this proposal is to give admins concrete advice on what 
they can do now and not in five or ten years from now.

>>> So PI is good for a group of participants limited in number - like,
>>> "medium sized and large businesses and service providers" (in the order
>>> of 10.000s to 100.000s), but is a clear no-go for "every end user that
>>> wants to have a slightly more robust Internet connection" (in the order
>>> of many millions).
>> Do keep in mind that said individual would have to deal with RIR
>> or LIR to get the PI space. I???m pretty sure that???s a no brainer
>> for a lot of people on this mailing list; I???m already too lazy
>> for that ;)
> There's service providers who help with that.  Like, me.
>
> So, if you throw money my way, I'll deal with the RIR paperwork (for
> RIPE NCC).
>
> Tell everbody "PI is the only answer and Gert will do the paperwork for
> you", and I can retire with lots of money very soon :-) - but I still
> think that this is not a good overall strategy.

And still getting and using a PI is generally cost prohibitive. So it 
will only be chosen by entities that have to. Originally when I 
encountered the multi-homing issues, I considered just getting a PI and 
call it a day. But in practice it is not that simple. So the pollution 
probably is not that big of an issue in practice (or I underestimate the 
risk for it significantly).

Maybe it helps the discussion to spill out the issues a SOHO currently 
faces when trying to use a PI to get away from the impression that it is 
way too easy and appealing:
* Where to get it? => LIR, but in practice? Either you know the people 
personally or you need one of a had full of people that offer it 
publicly (or a service provider like you).
* If I use a service provider that means it increases the costs, but 
let's say a home user is willing to spend it. OK.
* Now the next issue is how do I actually get traffic to my network at 
home using that prefix? I need to advertise it to the internet. But how 
do I do that? I need an ISP that speaks BGP with me and acts as my 
transit. So in practice I have two options. Either get to the most 
expensive internet service tier of my ISP or to use a cloud provider 
like Azure for example that offers this. => Either way all costs for 
this are "upon request" (I.E. almost all SOHO users and admins will 
consider it out of scope and too expensive). Also the technical tasks 
from here on are very advanced and require in depth networking 
knowledge. So another reason why SOHO won't be able to use it.
* In the first case that contract is not only required once, but for all 
up-links and for redundancy also from multiple ISPs. => But in practice 
for true redundancy now the SOHO admin needs to know about BGP, RPKI, 
... and also be able to debug traffic routing issues on a global scale. 
=> So now we're at a point where SOHO admins may sooner or later drop 
this approach because of the complexity, time investment and costs 
combined. Also this is at a more expensive tier than "just" static PA 
prefixes which are often at most only twice as much as the home user 
products and already the product small companies have. And then the 
solution using multiple PA-Prefixes is probably already more appealing.
* In the 2nd case where a cloud provider is used. The admin may not need 
to know about BGP, but thereby introduces the cloud provider as single 
point of failure. And if multiple cloud providers are used then the same 
issues as in the first case apply as as (at least to my knowledge) 
nobody is offering a "BGPaaS across multiple cloud providers". But for 
now ignore that part and say we're fine with only one cloud provider.
* So now we still cannot get traffic using our prefix out of the local 
network into the internet. We now need a tunnel into our cloud. So we 
deploy a VPN endpoint within our cloud and configure routing for it. => 
And when we now review this setup. It looks suspiciously like a typical 
LISP deployment but with way higher costs, complexity and administrative 
overhead. And because we have to rely on a single cloud provider in most 
cases anyway, why not use one of the static PA prefix that we can get 
from the cloud provider using a few clicks instead?
* But even if we want to use multiple cloud providers, then we would get 
multiple static PA prefixes. And because that doesn't require the in 
depth knowledge and capability of debugging traffic routing issues on 
the global internet, SOHO most likely naturally arrives at the 
conclusion of not using PI either way and chooses one of the other 
solutions instead.

*Tl;Dr:* So only a few SOHOs (to be realistic probably only those with 
experienced network admins that would do it anyway) would go with the PI 
solution and this proposal probably won't influence their decision much.

I think together with what Eduard already wrote that an explicit mention 
of the case with multiple static PAs within the PA solution will be 
enough. What do you think?

Sincerely,
Klaus Frank

> Gert Doering
>          -- NetMaster
>
> _______________________________________________
> v6ops mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/v6ops