RE: Questions about QUIC and Anycast

["Lubashev, Igor" <[e-mail]>]

> The details of how to deal with PTB messages is in https://quicwg.org/base-
> drafts/draft-ietf-quic-transport.html#packet-size
> 
> We realize that there is a limitation here: the PTB message won't contain the
> connection ID selected by the server, so it can get misrouted.  Kazuho
> suggested using a dummy packet with a long header on PMTUD probes for
> that reason (those packets can be skipped and contain both connection IDs),
> but I don't think that we've agreed to formalize that technique, so there
> might be some privacy implications we haven't considered.

Bill, note that this problem is not unique to Anycast (assuming that the ICMP PTB landed in the correct anycast cluster).  Any load balanced cluster that uses Connection IDs to route packets within the cluster will need to address this.

If the cluster is using a Consistent Hash algorithm to route the initial packets based on the 4-tuple of the packet, the ICMP can also be routed using the same hash.  Otherwise, one can keep state on the cluster's load balancer (active 4-tuples) or do rate-limited broadcast of the ICMP within the cluster.  None of the above (except for keeping centralized state) is a guarantee of success but does improve the chances.

If the Server's Connection ID is critical to you, you can use the long-header dummy packet coalesced with a short header packet trick for PMTU probe packets.  If a UDP packet containing these coalesced QUIC packets gets through, the long-header packet will be ignored but the short-header packet will be ACKed (just make sure to include at least one ACK-required frame -- a single PING frame will do the trick).  If that UDP packet elicits an ICMP PTB message, and that message contains more than 8 bytes of the original UDP datagram (the old IPv4 ICMP standard), the ICMP PTB message will contain your Server's Connection ID.


> -----Original Message-----
> From: Martin Thomson <[e-mail]>
> Sent: Monday, December 10, 2018 7:41 PM
> To: [e-mail]
> Cc: QUIC WG <[e-mail]>
> Subject: Re: Questions about QUIC and Anycast
> 
> Thanks for the questions Bill,
> 
> I'll provide some short answers.  Others might be able to expand a little.
> 
> On Tue, Dec 11, 2018 at 11:18 AM William Herrin <[e-mail]> wrote:
> > Question 1: Did I miss anything? Do the QUIC authors envision any
> > additional ways of supporting Anycast-routed servers?
> 
> The two methods you describe are what I understand to be the primary
> mechanisms.  You might find that connection ID-based routing (your second
> option) to be necessary though for the reasons discussed below...
> 
> > Question 2: Which packet contains the server preferred address?
> 
> In practice, taking some of the less-used exchanges (Retry, Version
> Negotiation, a TLS HelloRetryRequest) out of the picture, the server will
> often include the value in the first flight of packets it sends.
> However, that can be a couple of round trips away from the end of the
> handshake, because the handshake is multiple packets and might need to be
> repaired due to packet loss.
> 
> > Can the client start
> > trying to use the preferred address with its second packet or does it
> > have to wait?
> 
> The client has to wait, because - until the handshake is complete - it can't
> authenticated the preferred address value provided by the server.  Also,
> validation of the path to that address likely takes time.  That basically forces
> the server to use connection IDs to ensure continuity.  A server might rely on
> the route being stable for the duration of the handshake, though that would
> rely more on having timely termination of the connection; keeping in mind
> that our best mechanism, stateless reset, probably isn't viable at this point.
> 
> > Is it allowed to continue using the non-preferred address and, if so,
> > will it fail over to attempting the preferred address if its
> > communication fails in some way (no response or rejection from the
> > server at the non-preferred address)?
> 
> Yes, a client can choose to ignore the preferred address.  And yes, I believe
> that the working group understands the implications of this.
> If someone didn't and finds this distressing, yell.
> 
> > Question 3: What picks the connection ID? Does the API allow the
> > application software to influence the selection of connection ID in
> > any way?
> 
> The server picks connection IDs that the client puts in packets.  That selection
> really has to be done in consultation with load balancers or the folks
> managing routing configurations.
> 
> There's work on a protocol for managing this:
> https://tools.ietf.org/html/draft-duke-quic-load-balancers-03
> 
> > Question 4: Is the connection ID inside or outside the encrypted
> > portion of the packet?
> 
> The value is unencrypted, but it is authenticated.  So it can be read, but not
> changed.
> 
> > Question 5: Fragmentation needed / Packet Too Big messages originate
> > from an intermediate router that often has.a different return path
> > than the route taken by packets from the endpoint. How is QUIC
> > intended to deal with packet too big messages which arrive at a
> > different node in the Anycast cluster than the one which sent the
> > overlarge packet?
> 
> The details of how to deal with PTB messages is in https://quicwg.org/base-
> drafts/draft-ietf-quic-transport.html#packet-size
> 
> We realize that there is a limitation here: the PTB message won't contain the
> connection ID selected by the server, so it can get misrouted.  Kazuho
> suggested using a dummy packet with a long header on PMTUD probes for
> that reason (those packets can be skipped and contain both connection IDs),
> but I don't think that we've agreed to formalize that technique, so there
> might be some privacy implications we haven't considered.