Re: [v6ops] I-D Action: draft-jaeggli-v6ops-pmtud-ecmp-problem-00.txt

[joel jaeggli <[e-mail]>]

On 6/3/14, 6:18 PM, Brian E Carpenter wrote:
> Hi,
> 
>>    A problem common to the approach of distribution through hashing is
>>    its impact on path MTU discovery.  An ICMPv6 type 2 PTB message
>>    generated on the path between a client and an ECMP load balanced
>>    server will have the anycast address as the destination and will be
>>    statelessly load balanced to one of the anycast servers. 
> 
> This may seem picky, but I think the reader's brain will run
> more smoothly if it's explicit that it's a PTB triggered by
> a packet *from* the server and therefore directed *to* the server.
> "on the path" doesn't quite contain that information.

I think that's a reasonable observation and fairly straight forward to
adjust. To my mind the destination address being the anycast address
does communicate the direction of this ptb message.

>>             Because of this, the results of
>>    the ICMPv6 ECMP hash do not match that of the corresponding TCP or
>>    UDP ECMP hash.
> 
> Again picky, but if there are (say) 2 paths, there's a 50% chance
> that it gets the right path, etc. So "might not match" is probably
> better.

If there are 64 hash buckets the probability of ending up in the same
one is 1.5% if there are 255 it's ~.4%. so may not is potentially a near
certainty. It's entirely possible to have more hash buckets than
next-hops in order to facilitate load balancing without rehashing when
adding and removing devices in which case the distribution may vary, and
on those cases a packet hashed to the wrong bucket may well arrive on
the correct server/load-balancer.

> General comment: I'm not sure what is specific to ECMP
> about this problem. We identified it as a general (but out of
> scope) problem in RFC 7098. We did include this comment there:

So it may be out of scope for your rfc, I'm pretty sure that doesn 't
make the problem go away. I've been dinking around with large scale
load-balancing of this flavor for some time, more than a decade in on
form or another. We found it to be a commercial necessity to address the
problem of how to make PTB work as part of deploying consumer facing
internet applications services. it's not unique to ICMP, e.g. my
interest in the fragdrop discussion is due to similar problems.

>  o  Note that correct handling of ICMPv6 for Path MTU Discovery
>     requires the layer 3/4 balancer to keep state for the client
>     source address, independently of either the port numbers or the
>     flow label.

So there are two problems with this assertion.

1. state sharing is presently impossible at that scale I'm operating at.
so any solution that presumes it across an entire population of devices
isn't going to work. state sharing between pairs of devices in otherwise
stateless clusters (which describes the internal architecture of a
number of high-end firewalls and load-balancer products doesn't solve
this either).

2. the ptb packet doesn't have the parts of the flow associated within
the ip and icmp header so the flow that is was associated with cannot be
reconstructed by a pure l3/l4 device stateful or otherwise, it can as
noted in the draft be derived by inspecting the icmp payload for the
IP/TCP/UDP header of outgoing packet which is pretty deep packet
inspection (normally done by the end system). Finding the ICMP header is
not the same as being able to parse the payload. In any event we were
using rather high-end but but otherwise normal router silicon, so we get
to do this the same as anyone else who fronts a load-balancer or server
tier containing ecmp nexhops with a router...

> because, indeed, the PMTU is a function of the address pair only.

The ptb packet has the source address of the device that emitted it.

In any event, our goal isn't really to boil the ocean, with respect to
parsing the packet or deriving correct host, if we could get close
enough, it was to not break PMTUD, and therefore customers in a service
that supported millions of  users.

>     Brian
> 
> _______________________________________________
> v6ops mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/v6ops
>