Re: [v6ops] An Update to Happy Eyeballs

[Mark Andrews <[e-mail]>]

In message <[e-mail]>, David Schinazi wri
tes:
> Mark,
> 
> We have operational data proving that asynchronous DNS is not "for no
> good reason".  DNS queries fail rarely (it still happens) but they often
> can take several hundreds of milliseconds longer.
> The user shouldn't have to wait that delay for their webpage to load,
> and our data shows that this timer fires regularly in the field 
> indicating that users would have been needlessly waiting for their content.
> Why should we artificially delay the user's experience for an A response
> if we already have the AAAA?

So you now want Happy Eyeballs, which was designed to remove delays
of 10's - 100's of seconds before failover, to now override address
selection policies because the DNS's cache has one address type
present and not the other.  That is what the 50ms timer achieves.
It takes ~200ms to get a answer from the other side of the world
with terrestial links.

Happy Eyeballs was never about having the absolute fastest time to
connect.  It was about establishing a connection in a reasonable
amount of time in presence of network / server failures without
having ridiciously long failover delays.

Mark

> Erik,
> 
> Thanks for the suggestions!
> - we'll add a section about problems that Happy Eyeballs do not solve
> - regarding the increased retry timer, this is somewhat covered by
>   historical RTT information.
>   I don't think failures to reach the first address should harm the
>   next ones, as that defeats redundancy
> - I do agree that the main downside of Happy Eyeballs is hiding failures,
>   but enforcing clients have a reporting system seems uncommon for IETF
>   protocols, do you know of any?
> - Regarding TFO / TLS 1.3 0RTT I think the requirement is that you MUST
>   NOT use those if your data isn't idempotent, as packets could be
>   duplicated in the network. This is orthogonal to Happy Eyeballs.
> 
> Thanks,
> David Schinazi
> 
> 
> > On Mar 13, 2017, at 19:02, Erik Nygren <[e-mail]> wrote:
> > 
> > It's great to see an updated version of this guidance!
> > 
> > > There is zero reason for making async DNS a MUST.  
> > 
> > The bare minimum is that revolvers must do A and AAAA lookups in parallel.
> > The current behavior of some stacks is to do the AAAA and A lookups in seri
> es. 
> > This means effectively means that adding IPv6 connectivity to a client adds
>  an 
> > extra RTT in for almost all DNS lookups.  
> > For example, see section 5 in:  https://www.akamai.com/us/en/multimedia/doc
> uments/technical-publication/a-case-for-faster-mobile-web-in-cellular-ipv6-ne
> tworks.pdf <https://www.akamai.com/us/en/multimedia/documents/technical-publi
> cation/a-case-for-faster-mobile-web-in-cellular-ipv6-networks.pdf>
> > For clients such as mobile device visiting sites with lots of hostnames,
> > this can have a very substantial performance hit.  This also shows up
> > in some RUM-based measurement reports making IPv6 look slower 
> > (due to clients with IPv6 spending more time doing DNS lookups before 
> > doing page loads). 
> > 
> > Doing the lookups in parallel but not waiting for both responses is better 
> than serial,
> > but still has a perf hit (whatever the recovery time plus an RTT) when 
> > the A or AAAA lookup packet is lost.
> > 
> > Some additional comments/thoughts after reading through the -01 version:
> > 
> > * It would be good to add a section on failure cases NOT detected/mitigated
>  by this form of Happy Eyeballs.
> >    Even if not covering mitigations, it would still be good to discuss them
>  for awareness.
> > 
> >     - In particular, PMTUD seems to be the most common.  ie, if there is a 
> PMTUD issue between
> >       the client and server, then the connect will often succeed but the co
> nnection will fail to function.
> >       I think most large-scale IPv6 server operators (plus many of small on
> es) have broken this at least once.
> >       One client-side mitigation for TCP might be for the client to offer p
> rogressively smaller MSS 
> >       as it retries different IPs within a protocol family.
> >       (I don't know if anyone does or has tried this?  There is the server-
> side pmtud probing feature.)
> >       For UDP protocols, using full-frame packets for the SYN and the SYN-A
> CK (as QUIC does) seems
> >       like one approach to at least detect breakage early, although QUIC do
> esn't key have a PMTUD 
> >       mitigation solution AFAIK.
> > 
> >     -  Servers that return different content for IPv6 vs IPv4 (eg, "404 not
>  found" due to an unconfigured server on the IPv6 side).
> >        "Don't do this" as advice to server operators is likely the best way
>  to fix it as hacking around it on the client side is unhelpful.
> > 
> > * It may make sense to recommend some form of back-off in the retry timing.
>   Rather than a fixed value (eg, 250ms), adding
> >   an increasing time value with some jitter into each retry may be safer in
>  the cases of overloaded servers
> >   or a network connection that is borderline near the retry time.  I've see
> n congestive failure and lack-of-progress
> >   scenarios from having a fixed retry timer.  For example, with servers tha
> t do FIFO queueing of connections to accept,
> >   if the queue becomes longer than the retry period then all clients fail t
> o make forward progress and you reach 
> >   congestive collapse.  The same can happen with links that become high-lat
> ency, however.
> > 
> > * It may be worth adding some guidance into reporting and visibility.  I'm 
> not sure how?  
> >   One of the big complaints against Happy Eyeballs is that it masks brokenn
> ess (latency spikes
> >   but things keep working so no one complains enough to fix the root cause)
> .
> >   Having a recommendation that stacks or applications at least keep counter
> s and telemetry
> >   on failures may at least make it more viable to debug?
> > 
> > * It would be good to provide guidance or a reminder around protocols that 
> send data along with a SYN
> >   or an initial flight  (eg, TCP Fast Open / TFO and TLS 1.3 0RTT).  In par
> ticular, you likely
> >   want to send this ONLY on one connection attempt (eg, the IPv6 attempt?) 
> as otherwise 
> >   the operation may be executed twice by the server.  This may cause undue 
> server load
> >   and for apps/clients incorrectly using TFO or 0RTT for non-idempotent ope
> rations 
> >   it could cause duplicate actions. 
> > 
> > Thanks!  Erik
> > 
> > 
> > 
> > 
> > 
> > 
> > On Sun, Mar 12, 2017 at 11:53 PM, David Schinazi <[e-mail] <mail
> to:[e-mail]>> wrote:
> > Hi everyone,
> > 
> > Thanks a lot for the comments and feedback.
> > We've incorporated them into -01, please let us know if they were properly 
> addressed.
> > https://www.ietf.org/internet-drafts/draft-pauly-v6ops-happy-eyeballs-updat
> e-01.txt <https://www.ietf.org/internet-drafts/draft-pauly-v6ops-happy-eyebal
> ls-update-01.txt>
> > 
> > Regards,
> > David Schinazi
> > 
> > 
> >> On Mar 10, 2017, at 14:54, Philip Homburg <[e-mail] <mai
> lto:[e-mail]>> wrote:
> >> 
> >> In your letter dated Fri, 10 Mar 2017 09:29:55 -0800 you wrote:
> >>> We can certainly soften some of the language to make it clear that if 
> >>> your system has no such option, you are not necessarily out of spec, but 
> if su
> >>> ch an option is available, we believe that it SHOULD indeed be used. This
>  fits
> >>> with the Happy Eyeballs paradigm: if I am waiting for one of the DNS resp
> onse
> >>> s to come back, I could have already made my connection in that time, get
> ting 
> >>> the user the resource loaded more quickly.
> >> 
> >> If the DNS requirements can be toned down to the point that an application
> >> can use getaddrinfo if that fits the application, then that's fine
> >> with me.
> >> 
> >> 
> >> _______________________________________________
> >> v6ops mailing list
> >> [e-mail] <mailto:[e-mail]>
> >> https://www.ietf.org/mailman/listinfo/v6ops <https://www.ietf.org/mailman/
> listinfo/v6ops>
> > 
> > 
> > _______________________________________________
> > v6ops mailing list
> > [e-mail] <mailto:[e-mail]>
> > https://www.ietf.org/mailman/listinfo/v6ops <https://www.ietf.org/mailman/l
> istinfo/v6ops>
> > 
> > 
> 
> 
> --Boundary_(ID_F8KG9fRFqYtBb8MnlxB0dw)
> Content-type: text/html; CHARSET=US-ASCII
> Content-transfer-encoding: quoted-printable
> 
> <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
> charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
> -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
> class=3D""><div class=3D"">Mark,</div><div class=3D""><br =
> class=3D""></div><div class=3D"">We have operational data proving that =
> asynchronous DNS is not "for no good reason".</div><div class=3D"">DNS =
> queries fail rarely (it still happens) but they often can take several =
> hundreds of milliseconds longer.</div><div class=3D"">The user shouldn't =
> have to wait that delay for their webpage to load, and our data shows =
> that this timer</div><div class=3D"">fires regularly in the field =
> indicating that users would have been needlessly waiting for their =
> content.</div><div class=3D"">Why should we artificially delay the =
> user's experience for an A response if we already have the =
> AAAA?</div><div class=3D""><br class=3D""></div><div =
> class=3D"">Erik,</div><div class=3D""><br class=3D""></div><div =
> class=3D"">Thanks for the suggestions!</div><div class=3D"">- we'll add =
> a section about problems that Happy Eyeballs do not solve</div><div =
> class=3D"">- regarding the increased retry timer, this is somewhat =
> covered by historical RTT information.</div><div class=3D"">&nbsp; =
> &nbsp; I don't think failures to reach the first address should harm the =
> next ones, as that defeats redundancy</div><div class=3D"">- I do agree =
> that the main downside of Happy Eyeballs is hiding failures,</div><div =
> class=3D"">&nbsp; &nbsp; but enforcing clients have a reporting system =
> seems uncommon for IETF protocols, do you know of any?</div><div =
> class=3D"">- Regarding TFO / TLS 1.3 0RTT I think the requirement is =
> that you MUST NOT use those</div><div class=3D"">&nbsp; &nbsp; if your =
> data isn't idempotent, as packets could be duplicated in the network. =
> This is orthogonal to Happy Eyeballs.</div><div class=3D""><br =
> class=3D""></div><div class=3D"">Thanks,</div><div class=3D"">David =
> Schinazi</div><div class=3D""><br class=3D""></div><br =
> class=3D""><div><blockquote type=3D"cite" class=3D""><div class=3D"">On =
> Mar 13, 2017, at 19:02, Erik Nygren &lt;<a =
> href=3D"mailto:[e-mail]" =
> class=3D"">[e-mail]</a>&gt; wrote:</div><br =
> class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" =
> class=3D""><div class=3D""><div class=3D""><div class=3D""><div =
> class=3D""><div class=3D""><div class=3D""><div class=3D""><div =
> class=3D""><div class=3D"">It's great to see an updated version of this =
> guidance!<br class=3D""><br class=3D"">&gt; There is zero reason for =
> making async DNS a MUST.&nbsp; <br class=3D""><br class=3D""></div><div =
> class=3D"">The bare minimum is that revolvers must do A and AAAA lookups =
> in parallel.<br class=3D""></div><div class=3D"">The current behavior of =
> some stacks is to do the AAAA and A lookups in series. <br class=3D"">This=
>  means effectively means that adding IPv6 connectivity to a client adds =
> an <br class=3D""></div><div class=3D"">extra RTT in for almost all DNS =
> lookups.&nbsp; <br class=3D""></div><div class=3D"">For example, see =
> section 5 in:&nbsp; <a =
> href=3D"https://www.akamai.com/us/en/multimedia/documents/technical-public=
> ation/a-case-for-faster-mobile-web-in-cellular-ipv6-networks.pdf" =
> class=3D"">https://www.akamai.com/us/en/multimedia/documents/technical-pub=
> lication/a-case-for-faster-mobile-web-in-cellular-ipv6-networks.pdf</a><br=
>  class=3D""></div><div class=3D"">For clients such as mobile device =
> visiting sites with lots of hostnames,<br class=3D""></div><div =
> class=3D"">this can have a very substantial performance hit.&nbsp; This =
> also shows up<br class=3D""></div><div class=3D"">in some RUM-based =
> measurement reports making IPv6 look slower <br class=3D""></div><div =
> class=3D"">(due to clients with IPv6 spending more time doing DNS =
> lookups before <br class=3D""></div><div class=3D"">doing page loads). =
> <br class=3D""><br class=3D"">Doing the lookups in parallel but not =
> waiting for both responses is better than serial,<br class=3D""></div><div=
>  class=3D"">but still has a perf hit (whatever the recovery time plus an =
> RTT) when <br class=3D"">the A or AAAA lookup packet is lost.<br =
> class=3D""></div><div class=3D""><br class=3D""></div>Some additional =
> comments/thoughts after reading through the -01 version:<br class=3D""><br=
>  class=3D""></div>* It would be good to add a section on failure cases =
> NOT detected/mitigated by this form of Happy Eyeballs.<br =
> class=3D""></div><div class=3D"">&nbsp;&nbsp; Even if not covering =
> mitigations, it would still be good to discuss them for awareness.<br =
> class=3D""></div><br class=3D"">&nbsp;&nbsp;&nbsp; - In particular, =
> PMTUD seems to be the most common.&nbsp; ie, if there is a PMTUD issue =
> between<br class=3D""></div>&nbsp; &nbsp; &nbsp; the client and server, =
> then the connect will often succeed but the connection will fail to =
> function.<br class=3D""></div><div =
> class=3D"">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; I think most large-scale IPv6 =
> server operators (plus many of small ones) have broken this at least =
> once.<br class=3D""></div>&nbsp;&nbsp; &nbsp;&nbsp; One client-side =
> mitigation for TCP might be for the client to offer progressively =
> smaller MSS <br class=3D"">&nbsp;&nbsp; &nbsp;&nbsp; as it retries =
> different IPs within a protocol family.<br =
> class=3D""></div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (I don't know if anyone =
> does or has tried this?&nbsp; There is the server-side pmtud probing =
> feature.)<br class=3D""></div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; For UDP =
> protocols, using full-frame packets for the SYN and the SYN-ACK (as QUIC =
> does) seems<br class=3D""></div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; like one =
> approach to at least detect breakage early, although QUIC doesn't key =
> have a PMTUD <br class=3D""></div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
> mitigation solution AFAIK.<br class=3D""><div class=3D""><div =
> class=3D""><div class=3D""><div class=3D""><div class=3D""><div =
> class=3D""><div class=3D""><div class=3D""><br class=3D""><div =
> class=3D"">&nbsp;&nbsp;&nbsp; -&nbsp; Servers that return different =
> content for IPv6 vs IPv4 (eg, "404 not found" due to an unconfigured =
> server on the IPv6 side).<br class=3D""></div><div =
> class=3D"">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; "Don't do this" as =
> advice to server operators is likely the best way to fix it as hacking =
> around it on the client side is unhelpful.<br class=3D""></div><div =
> class=3D""><br class=3D""></div><div class=3D"">* It may make sense to =
> recommend some form of back-off in the retry timing.&nbsp; Rather than a =
> fixed value (eg, 250ms), adding<br class=3D""></div><div class=3D"">&nbsp;=
>  an increasing time value with some jitter into each retry may be safer =
> in the cases of overloaded servers<br class=3D""></div><div =
> class=3D"">&nbsp; or a network connection that is borderline near the =
> retry time.&nbsp; I've seen congestive failure and lack-of-progress<br =
> class=3D""></div><div class=3D"">&nbsp; scenarios from having a fixed =
> retry timer.&nbsp; For example, with servers that do FIFO queueing of =
> connections to accept,<br class=3D""></div><div class=3D"">&nbsp; if the =
> queue becomes longer than the retry period then all clients fail to make =
> forward progress and you reach <br class=3D""></div><div class=3D"">&nbsp;=
>  congestive collapse.&nbsp; The same can happen with links that become =
> high-latency, however.<br class=3D""></div><div class=3D""><br =
> class=3D""></div><div class=3D"">* It may be worth adding some guidance =
> into reporting and visibility.&nbsp; I'm not sure how?&nbsp; <br =
> class=3D""></div><div class=3D"">&nbsp; One of the big complaints =
> against Happy Eyeballs is that it masks brokenness (latency spikes<br =
> class=3D""></div><div class=3D"">&nbsp; but things keep working so no =
> one complains enough to fix the root cause).<br class=3D""></div><div =
> class=3D"">&nbsp; Having a recommendation that stacks or applications at =
> least keep counters and telemetry<br class=3D""></div><div =
> class=3D"">&nbsp; on failures may at least make it more viable to =
> debug?<br class=3D""><br class=3D""></div><div class=3D"">* It would be =
> good to provide guidance or a reminder around protocols that send data =
> along with a SYN<br class=3D"">&nbsp; or an initial flight&nbsp; (eg, =
> TCP Fast Open / TFO and TLS 1.3 0RTT).&nbsp; In particular, you =
> likely<br class=3D"">&nbsp; want to send this ONLY on one connection =
> attempt (eg, the IPv6 attempt?) as otherwise <br class=3D""></div><div =
> class=3D"">&nbsp; the operation may be executed twice by the =
> server.&nbsp; This may cause undue server load<br class=3D""></div><div =
> class=3D"">&nbsp; and for apps/clients incorrectly using TFO or 0RTT for =
> non-idempotent operations <br class=3D""></div><div class=3D"">&nbsp; it =
> could cause duplicate actions. <br class=3D""></div><div class=3D""><br =
> class=3D""></div><div class=3D"">Thanks!&nbsp; Erik<br class=3D""><br =
> class=3D""></div><div class=3D""><br class=3D""><br class=3D""><br =
> class=3D""><br =
> class=3D""></div></div></div></div></div></div></div></div></div></div><di=
> v class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On =
> Sun, Mar 12, 2017 at 11:53 PM, David Schinazi <span dir=3D"ltr" =
> class=3D"">&lt;<a href=3D"mailto:[e-mail]" target=3D"_blank" =
> class=3D"">[e-mail]</a>&gt;</span> wrote:<br =
> class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
> .8ex;border-left:1px #ccc solid;padding-left:1ex"><div =
> style=3D"word-wrap:break-word" class=3D"">Hi everyone,<div class=3D""><br =
> class=3D""></div><div class=3D"">Thanks a lot for the comments and =
> feedback.</div><div class=3D"">We've incorporated them into -01, please =
> let us know if they were properly addressed.</div><div class=3D""><a =
> href=3D"https://www.ietf.org/internet-drafts/draft-pauly-v6ops-happy-eyeba=
> lls-update-01.txt" target=3D"_blank" =
> class=3D"">https://www.ietf.org/internet-<wbr =
> class=3D"">drafts/draft-pauly-v6ops-<wbr =
> class=3D"">happy-eyeballs-update-01.txt</a></div><div class=3D""><br =
> class=3D""></div><div class=3D"">Regards,</div><div class=3D"">David =
> Schinazi</div><div class=3D""><div class=3D"h5"><div class=3D""><br =
> class=3D""></div><div class=3D""><br class=3D""><div =
> class=3D""><blockquote type=3D"cite" class=3D""><div class=3D"">On Mar =
> 10, 2017, at 14:54, Philip Homburg &lt;<a =
> href=3D"mailto:[e-mail]" target=3D"_blank" =
> class=3D"">[e-mail]</a>&gt; wrote:</div><br =
> class=3D"m_5032583941560406556Apple-interchange-newline"><div =
> class=3D""><div class=3D"">In your letter dated Fri, 10 Mar 2017 =
> 09:29:55 -0800 you wrote:<br class=3D""><blockquote type=3D"cite" =
> class=3D"">We can certainly soften some of the language to make it clear =
> that if <br class=3D"">your system has no such option, you are not =
> necessarily out of spec, but if su<br class=3D"">ch an option is =
> available, we believe that it SHOULD indeed be used. This fits<br =
> class=3D"">with the Happy Eyeballs paradigm: if I am waiting for one of =
> the DNS response<br class=3D"">s to come back, I could have already made =
> my connection in that time, getting <br class=3D"">the user the resource =
> loaded more quickly.<br class=3D""></blockquote><br class=3D"">If the =
> DNS requirements can be toned down to the point that an application<br =
> class=3D"">can use getaddrinfo if that fits the application, then that's =
> fine<br class=3D"">with me.<br class=3D""><br class=3D""><br =
> class=3D"">______________________________<wbr =
> class=3D"">_________________<br class=3D"">v6ops mailing list<br =
> class=3D""><a href=3D"mailto:[e-mail]" target=3D"_blank" =
> class=3D"">[e-mail]</a><br class=3D""><a =
> href=3D"https://www.ietf.org/mailman/listinfo/v6ops" target=3D"_blank" =
> class=3D"">https://www.ietf.org/mailman/<wbr =
> class=3D"">listinfo/v6ops</a><br =
> class=3D""></div></div></blockquote></div><br =
> class=3D""></div></div></div></div><br =
> class=3D"">______________________________<wbr =
> class=3D"">_________________<br class=3D"">
> v6ops mailing list<br class=3D"">
> <a href=3D"mailto:[e-mail]" class=3D"">[e-mail]</a><br =
> class=3D"">
> <a href=3D"https://www.ietf.org/mailman/listinfo/v6ops" rel=3D"noreferrer"=
>  target=3D"_blank" class=3D"">https://www.ietf.org/mailman/<wbr =
> class=3D"">listinfo/v6ops</a><br class=3D"">
> <br class=3D""></blockquote></div><br class=3D""></div>
> </div></blockquote></div><br class=3D""></body></html>=
> 
> --Boundary_(ID_F8KG9fRFqYtBb8MnlxB0dw)--
> 
> 
> --===============7303905125349802389==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
> 
> _______________________________________________
> v6ops mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/v6ops
> 
> --===============7303905125349802389==--
> 
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: [phone]                 INTERNET: [e-mail]