Re: [v6ops] [IPv6] New Version Notification for draft-fbnvv-v6ops-site-multihoming-01.txt

[Vasilenko Eduard <[e-mail]>]

Hi Mark,
Big thanks for this discussion.
You are right that the local interface would not help too much in MHMP designs.

I did not argue that the application may resolve the MHMP problem - it could make bind() before connect().
Just the application on the client side should have a rather complex logic (comparable to RFC 6724) that it typically does not have.
You are much more experienced on the application side. Do you know examples when the application on the client side chooses the source address first?

Please, read section 5.2 https://datatracker.ietf.org/doc/html/draft-fbnvv-v6ops-site-multihoming-01#page-14
And propose improvements.

If we would add to the problem statement:
" It is important to mention that the topology in Figure 1 or all Figures of [MHMP Enterprise] assume redundant Carriers' connection to the router upstream. Direct Carrier connection to the host (for example 3GPP modem) is not in the scope of this document that has “site connection” in the name."

And to section 5.2.3:
" The host stack (in [MPTCP] case) or the host application (in [MPQUIC] case) may resolve the MHMP problem in the PA environment by choosing the source address first (using bind()) as discussed in section 5.2.1. [MPTCP] socket interface extensions explain in sections 5.3.1 or 5.3.3 how to bind many local addresses to the future or current sessions. [MPQUIC] does not have a clear discussion on binding to local addresses but the [MPQUIC] in general transfers the majority of tasks to the application layer where it is possible."

Is it enough? (to touch MPTCP and MPQUIC)

Eduard
-----Original Message-----
From: Mark Smith [mailto:[e-mail]] 
Sent: Wednesday, July 12, 2023 1:11 PM
To: Vasilenko Eduard <[e-mail]>
Cc: Nick Buraglio <[e-mail]>; v6ops list <[e-mail]>; 6MAN <[e-mail]>
Subject: Re: [v6ops] [IPv6] New Version Notification for draft-fbnvv-v6ops-site-multihoming-01.txt

Hi Eduard,

On Wed, 12 Jul 2023 at 16:52, Vasilenko Eduard <[e-mail]> wrote:
>
> Hi Mark,
>
> MPTCP design goals: it MUST be transparent (not visible) for the application.
> Hence, they put in the document:
> "MPTCP MUST NOT be used for applications that request to bind to a specific address or interface, since such applications are making a deliberate choice of path in use ".
> Hence, it could not help a host in solving MHMP problems with PA addresses. The great majority of client applications use getaddrinfo() to get a random next hop and then (as a consequence) a random source IP address.

You're ascribing many more functions to getaddrinfo() and the applications than what actually occurs.

getaddrinfo() only returns a set of destination addresses, in the order specified by either RFC 3484 or RFC 6724, possibly influenced by a local host configuration such as in /etc/gai.conf under Linux.

---
mark@opex gai]$ ./gai www.ietf.org

getaddrinfo(www.ietf.org)

AF_INET6, 2606:4700::6810:2d63, sin6_scope_id = 0 AF_INET6, 2606:4700::6810:2c63, sin6_scope_id = 0 AF_INET, 104.16.45.99 AF_INET, 104.16.44.99

[mark@opex gai]$
---

Applications should then walk through the set of DAs, attempting to
connect() to them, until one is successful, as also described in RFC
3484 or RFC 6724 (2. Context in Which the Algorithms Operate). The Client Program example code on the glibc getaddrinfo() manual page demonstrates this - https://man7.org/linux/man-pages/man3/getaddrinfo.3.html - starting at
"/* Obtain address(es) matching host/port. */".

Source address selection is performed by the network stack below the application. The application asks the network stack to connect to the DA. The network stack then performs source address selection based on RFC 3484 or 6724.

This text in your ID only occurs when the packet, which already has a DA and SA chosen, triggers Neighbor Discovery for a next hop router, if there isn't a Neighbor Cache entry for the router.

By
   default, the next hop is chosen in a random round-robin manner
   between all available routers on the particular link (under [ND]
   section 6.3.6).

> MPTCP had in mind the situation when the 3GPP interface is directly connected to the host.

That was one of the use cases. It isn't the only one. Aggregating multiple interfaces' bandwidth is another one, as an alternative to ECMP or LAG.

>In such a situation MHMP problem is resolved automatically, even without MPTCP.
>

That's not correct. How can conventional single path TCP spread traffic over both a 3G and Wifi interface concurrently?

> I am waiting for the information on what is used by MP QUIC implementations: getaddrinfo() or bind() to establish a connection.
> In one case, it helps to resolve MHMP (not fully), in another case, it does not.
> IMHO: it is highly possible that MP QUIC implementations again assume the separate 3GPP interface on the host which makes it not relevant to the MHMP PA problem.
>

At the IP layer and higher, there is no difference between a host with
2 physical interfaces, each with a single address, or a host with a single physical interface, with multiple addresses. An IPv6 host with addresses from multiple PA address spaces from two different ISPs on a single interface is multihomed.

Regards,
Mark.


> Eduard
> -----Original Message-----
> From: Nick Buraglio [mailto:[e-mail]]
> Sent: Tuesday, July 11, 2023 8:21 PM
> To: Mark Smith <[e-mail]>
> Cc: Vasilenko Eduard <[e-mail]>; v6ops list 
> <[e-mail]>; 6MAN <[e-mail]>
> Subject: Re: [v6ops] [IPv6] New Version Notification for 
> draft-fbnvv-v6ops-site-multihoming-01.txt
>
>
>
>
> ------- Original Message -------
> On Tuesday, July 11th, 2023 at 10:11 AM, Mark Smith <[e-mail]> wrote:
>
>
> >
> >
> > On Tue, 11 July 2023, 16:57 Vasilenko Eduard, <[e-mail]> wrote:
> >
> > > Hi Brian,
> > > It is a theoretical "good wish". Practically, MPTCP RFC blocks 
> > > this possibility by the statement below, MPTCP in general has the goal to be transparent for the application.
> > > Hence, MPTCP is not capable in principle to solve any MHMP issues.
> >
> >
> > I don't understand this conclusion or assertion at all.
> >
> > The ideal is that network and host multihoming is transparent to applications.
> >
> > We don't want to burden application developers with having to customise their applications to suit multihomed networks or hosts.
> >
> > The advantage of MPTCP or MPQUIC is the multipathing, providing potentially both increased throughput and robustness is that those capabilities are hidden in the transport layer.
> >
> > (And the fundamental problem that makes network multihoming hard is that IP addresses have been used as transport layer connection identifiers, such that a transport layer connection can't continue if the underlying IP address it is bound to stops working - the locator/identifier problem.
> >
> > MPTCP and MPQUIC create their own connection identifiers, making the connections far more independent of the IP addresses used to send the packets between the endpoints.
> >
> > TCP probably should have got it's own connection identifiers when it 
> > was split off from IP in the 1970s.)
>
> I really hope I am wrong about this, having another multi-homing option to try for a site would be a really fun experiment. I will re-read rfc8684 with more focus, but a brief skim looked like the MP-TCP spec relies on standard TCP setup signaling. My mind immediately goes to "how does this work for hosts with dual PA provided addressing?" and "how does this work for the underlying network?". For MPTCP to work from a site perspective, it would need the hosts to implement this uniformly to be consistently supported, I assert, and the network would also need to understand the diverse routing of two PA, a PI and a PA, or two PI addresses on a given host, so we're - I think - back to the problem of different addressing and load sharing.
>
>
> I understand this probably isn't the only use case, and as always, happy to be wrong.
> >
> >
> >
> > >
> > > I strongly suspect that QUIC real applications are not mangling with source IPv6 addresses - they use getaddrinfo() with the random next hop and source address selection.
> >
> >
> >
> > You're guessing this.
> >
> >
> > > Albeit, QUIC implementation of multipathing assumes that the application should be developed specially (MP QUIC by design is not transparent for the application layer).
> >
> >
> > And then coming to a conclusion based on your own guess you've just made.
> >
> >
> > > Hence, QUIC is capable potentially to implement the logic for choosing the source address first (by bind()).
> > >
> > > Is anybody knows what the particular MP QUIC implementation is using for connection: getaddrinfo() or bind()?
> > > If any use bind() then it would resolve the most important MHMP 
> > > issues (not all - see section 5.2.1)
> > >
> > > Eduard
> > > -----Original Message-----
> > > From: Brian E Carpenter [mailto:[e-mail]]
> > > Sent: Tuesday, July 11, 2023 7:12 AM
> > > To: Vasilenko Eduard <[e-mail]>; Mark Smith 
> > > <[e-mail]>
> > > Cc: [e-mail]; [e-mail]
> > > Subject: Re: [IPv6] New Version Notification for 
> > > draft-fbnvv-v6ops-site-multihoming-01.txt
> > >
> > >
> > >
> > > Regards
> > > Brian Carpenter
> > >
> > > On 10-Jul-23 19:27, Vasilenko Eduard wrote:
> > > > Hi Mark,
> > > > Thanks a lot for this message. I would discuss it with co-authors, but IMHO: we need to mention MPTCP and QUIC Multipath. Many may have the same illusion as you.
> > > >
> > > > I. Let's consider the situation when we have only one interface on the host. Multiple Carriers are connected upstream (on the CPE?).
> > > >
> > > > About what is chosen first - the next hop or the source address:
> > > > MPTCP (rfc6182) has strict in 2 places: " MPTCP MUST NOT be used for applications that request to bind to a specific address or interface, since such applications are making a deliberate choice of path in use ".
> > > > QUIC Multipath (draft-ietf-quic-multipath) has nothing about binding (or not binding) to a specific source address or interface.
> > > >
> > > > Hence, it looks like they both would probably use getaddrinfo() that would randomly choose the next hop and then randomly again source address (as explained in section 5.2 of draft-fbnvv-v6ops-site-multihoming) which does not help in multihoming, not at all.
> > >
> > > Multipath means: try all the available paths. If you have N source 
> > > addresses and M destination addresses, that means N*M paths are 
> > > available. The order of probing is of course a good question. I 
> > > have no knowledge of QUIC but there's a little about MPTCP (and 
> > > SHIM6) in this expired draft and some of its references:
> > > https://datatracker.ietf.org/doc/html/draft-naderi-ipv6-probing-01
> > >
> > > Brian
> > >
> > >
> > >
> > > >
> > > > Do you know of any MPTCP or QUIC Multipath implementation that uses bind() initially (not getaddrinfo()) to reverse the logic (choose source address before the next hop)?
> > > > Then it would be a big deal that MUST be mentioned.
> > > >
> > > > Anyway, section 5.2 explains the situation properly for both cases (getaddrinfo() or bind()).
> > > > Just we need to mention that depending on what is chosen first for Multipathing solutions (next hop or source address) would drive the case to section 5.2.1 or 5.2.2.
> > > > Bind() (section 5.2.1) would have much more help to MHMP than getaddrinfo() (section 5.2.2).
> > > >
> > > > II. Let's consider the situation when we have many interfaces on the host. And at least one carrier is connected directly to the host (3GPP Modem?).
> > > >
> > > > Indeed, in this case, it does not matter what would be chosen first: the next hop or source address, getaddrinfo() would lead to the same result as Bind().
> > > > MHMP problem is resolved automatically. Does not matter whether MPTCP is used or not.
> > > > IMHO: we have lost the message in the draft that the host *directly connected* (by a separate interface) to at least one carrier has no MHMP problem in principle.
> > > >
> > > > Eduard
> > > > -----Original Message-----
> > > > From: Mark Smith [mailto:[e-mail]]
> > > > Sent: Monday, July 10, 2023 1:24 AM
> > > > To: Vasilenko Eduard <[e-mail]>
> > > > Cc: [e-mail]
> > > > Subject: Re: [IPv6] New Version Notification for 
> > > > draft-fbnvv-v6ops-site-multihoming-01.txt
> > > >
> > > > Hi,
> > > >
> > > > There's one multihoming solution that hasn't been mentioned in the draft that has been very widely deployed for both IPv4 and IPv6.
> > > >
> > > > Multipath TCP, deployed on Apple's IOS 7, and used by Siri.
> > > >
> > > > https://perso.uclouvain.be/olivier.bonaventure/blog/html/2013/09
> > > > /1
> > > > 8/mp
> > > > tcp.html
> > > >
> > > > https://blog.apnic.net/2022/08/23/analyzing-mptcp-adoption-in-th
> > > > e-
> > > > inte
> > > > rnet/
> > > >
> > > > QUIC is gaining multipath capabilities as well.
> > > >
> > > > https://multipath-quic.org/
> > > >
> > > > The only issue left for a multi-homed network is to steer traffic out of the correct ISP connection based on source address so that the packets don't get dropped by BCP 38 filters. Source routing, segment routing or simply IP tunnelling from the first hop router directly to the correct ISP egress edge router via addition of an IP in IP or GRE header solves that.
> > > >
> > > > Regards,
> > > > Mark.
> > > >
> > > >
> > > >
> > > > On Fri, 7 Jul 2023 at 19:34, Vasilenko Eduard <vasilenko.eduard=[e-mail]> wrote:
> > > >>
> > > >> Hi IPv6 WG,
> > > >>
> > > >> We have heavily edited the draft between versions 00 and 01 (216 commits on github).
> > > >>
> > > >> Primarily, we have expressed the general WG opinion in different sections on "how bad NAT and NPT are", propagating the hate to the table of content, including mentioning that NPT is experimental.
> > > >> Yet, we have not deleted solutions that exist in the real world - like we or not.
> > > >>
> > > >> The section on another "not good" solution is added - unfortunately, "application proxy" is very popular in Enterprise that may address the draft subject.
> > > >> Again, we were trying to be very negative on it, yet balanced - it is part of the real Enterprise practice and needs to be overviewed.
> > > >>
> > > >> More places to mention PVD when applicable.
> > > >>
> > > >> Paolo Nero has discovered that Microsoft is capable to use "preference" from RA even if RFC 4191 is not claimed.
> > > >>
> > > >> And many other small changes addressing comments from Tim 
> > > >> Winters, Eric Vincke, Toerless Eckert, Lorenzo Colitti, Sheng Jiang, Erik Nygren, Fernando Gont, Jared Mauch, and Juliusz Chroboczek.
> > > >> Thanks for the feedback!
> > > >>
> > > >> Eduard
> > > >> -----Original Message-----
> > > >> From: [e-mail] 
> > > >> [mailto:[e-mail]]
> > > >> Sent: Friday, July 7, 2023 11:58 AM
> > > >> To: Vasilenko Eduard <[e-mail]>; Vasilenko 
> > > >> Eduard <[e-mail]>; Klaus Frank 
> > > >> <[e-mail]>; Nick Buraglio 
> > > >> <[e-mail]>; Paolo Nero 
> > > >> <[e-mail]>; Paolo Volpato <[e-mail]>
> > > >> Subject: New Version Notification for 
> > > >> draft-fbnvv-v6ops-site-multihoming-01.txt
> > > >>
> > > >>
> > > >> A new version of I-D, draft-fbnvv-v6ops-site-multihoming-01.txt
> > > >> has been successfully submitted by Eduard Vasilenko and posted to the IETF repository.
> > > >>
> > > >> Name: draft-fbnvv-v6ops-site-multihoming
> > > >> Revision: 01
> > > >> Title: IPv6 Site connection to many Carriers Document date:
> > > >> 2023-07-07
> > > >> Group: Individual Submission
> > > >> Pages: 38
> > > >> URL:
> > > >> https://www.ietf.org/archive/id/draft-fbnvv-v6ops-site-multihom
> > > >> in
> > > >> g-01.txt
> > > >> Status:
> > > >> https://datatracker.ietf.org/doc/draft-fbnvv-v6ops-site-multiho
> > > >> mi
> > > >> ng/
> > > >> Htmlized:
> > > >> https://datatracker.ietf.org/doc/html/draft-fbnvv-v6ops-site-mu
> > > >> lt
> > > >> ihoming
> > > >> Diff:
> > > >> https://author-tools.ietf.org/iddiff?url2=draft-fbnvv-v6ops-sit
> > > >> e-
> > > >> multihoming-01
> > > >>
> > > >> Abstract:
> > > >> Carrier resilience is a typical business requirement. IPv4 
> > > >> deployments have traditionally solved this challenge through 
> > > >> private internal site addressing in combination with separate 
> > > >> NAT engines attached to multiple redundant carriers. IPv6 
> > > >> brings support for true end-to-end connectivity on the 
> > > >> Internet, and hence NAT is the least desirable option in such deployments.
> > > >> Native IPv6 solutions for carrier resiliency, however, have 
> > > >> drawbacks. This document discusses all currently-available 
> > > >> options to organize carrier resiliency for a site, their 
> > > >> strengths and weaknesses, and provides a history of past IETF 
> > > >> efforts approaching the issue. The views presented here are the 
> > > >> summary of discussions on the v6ops mailing list and are not just the personal opinion of the authors.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> The IETF Secretariat
> > > >>
> > > >>
> > > >>
> > > >> ---------------------------------------------------------------
> > > >> --
> > > >> --- IETF IPv6 working group mailing list [e-mail] 
> > > >> Administrative Requests:
> > > >> https://www.ietf.org/mailman/listinfo/ipv6
> > > >> ---------------------------------------------------------------
> > > >> --
> > > >> ---
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > -- IETF IPv6 working group mailing list [e-mail] 
> > > > Administrative Requests:
> > > > https://www.ietf.org/mailman/listinfo/ipv6
> > > > ----------------------------------------------------------------
> > > > --
> > > > --