Re: [IPsec] draft-xu-ipsecme-esp-in-udp-lb-07

[Antony Antony <[e-mail]>]

On Wed, Mar 31, 2021 at 23:38:01 +0000, Bottorff, Paul wrote:
> Hi Antony:
> 
> Below,
> 
> Cheers,
> 
> Paul
> 
> 
> 
> -----Original Message-----
> From: IPsec [mailto:[e-mail]] On Behalf Of Antony Antony
> Sent: Wednesday, March 31, 2021 3:32 AM
> To: Bottorff, Paul <[e-mail]>; IPsec <[e-mail]>
> Cc: [e-mail]
> Subject: Re: [IPsec] draft-xu-ipsecme-esp-in-udp-lb-07
> 
> Hi,
> 
> This is an interesting draft. I would love to see a generic solution for network paths and receiver use cases, such as RSS.
> 
> PB>> Can you explain your use case for RSS a little more? I'd guess you are looking at LB around the RSS queues to get better distribution for the decodes.
> <<

We are using muliple SAs, with identical Traffic Selectors, to
load balance IPsec traffic between IPsec peers.
https://datatracker.ietf.org/doc/draft-pwouters-multi-sa-performance/

This should improve multi-core-cpu utilization and IPsec throughput. The receiver
should distribute the flows, based on SPI, to different CPUs for decryption.
Ideally the NIC should support a ntuple flow distribution using SPI.
It is not yet widely supported, and also not compatible with older NICs.
Now we are thinking of ESP in UDP with different source ports. Just like the draft
proposed for ECMP and LAG.
When implementing it I ran into issues with NAT and SAD remapping messages which update IKE.
I am using strongSWAN for IKE and Linux XFRM for ESP.

> The RFC3948 specifies one pair of UDP ports 4500-4500.
> Both the IKE flow and the ESP in UDP flow should use the same UDP flow.
> The draft seems to suggest new destination port and source ports are only for ESP? How would this change work with IKE?
> May you are not intending to use IKE?

PB> Our use cases use IKE, however as stated in RFC3948 ESPinUDP does not have to be
PB> tied to IKE, it is just advantageous to do so for the NAT case since this allows
PB> a single mapping for both at the NAT rather than two mappings.


PB> I've wondered why we could not use the RFC3948 encoding for ESPinUDP, but allow 
PB> the source port to be chosen differently than IKE. Perhaps Xu has some thoughts on this. 

In my experience it would work well when there is no NAT. When there
there is NAT the IKE and ESP in UDP should use same ports, otherwise
IKE will get established and ESP packets could get dropped in one
direction. When there is NAT it would look more like a client-to-server
than a peer-to-peer.

> How would the new ESP flow work when there is a NAT gateway along the path?
> I ran into issues with both sides choosing different source ports.
> It would cause SADB mapping changes which force changes IKE flows. One coul disable
> SADB mapping changes. However, that would break real NAT changes.
>
PB> We are mostly interested in data centre use cases which don't have intervening NATs,
PB> however I believe SD-WAN cases could have NAT and FW traversals between tunnel end
PB> points. As it stands draft-xu-ipsecme-esp-in-udp-lb does not specify how the source port PB> value is determined. It seems like it could be based on a hash value within the ESP or
PB> based on the SPI and IPs.

I implemented a proof concept - UDP source port set to 16bit hash of out going SPI.
Then it occurred to me the hash could collide with other well
known UDP ports e.g. hash of an SPI or/and IP address could be 53(DNS).
Some admins may not be happy to see source port 53 for ESP in UDP traffic.

If you choose ephemeral source ports both sides should know the source
ports otherwise SADB remap will be generated. If we disable that real NAT
would break...
So far I don't have a clean solution which would work with NAT and without NAT
for our use case.