Re: [IPsec] Discussion of draft-pwouters-ipsecme-multi-sa-performance

[Paul Wouters <[e-mail]>]

On Tue, 18 Oct 2022, Valery Smyslov wrote:

>> I think this is implementation specific. You could install an temporary
>> rule into the SPD that would give the fallback SA more priority than the
>> per-CPU policy installed, so it wouldn't generate ACQUIRES for a while.
>
> Why for a while? And for how long? There is no indication
> from the peer whether inability to create more SAs is temporary
> or permanent, so we may end up with wasting resources when one peer will be constantly
> trying to install per-CPU SA with no success.

That's why we propose the hint of NUM_QUEUES :)
And why we have:

    The TS_MAX_QUEUE notify conveys that the peer is unwilling to create more
    additional Child SAs for this particular Traffic Selector set.

Each peer knows the amount of this child SA they have. So if they
receive a few Delete/Notifies perhaps than they can install more again
if needed.

>> Perhaps userland could also decide to terminate another per-CPU SA that
>> is idle. Although I think the advised policy is stated to install at
>> least one per-CPU SA per CPU (and allow a few more to catch any race
>> conditions and rekeys).
>
> Why more SAs than the number of CPUs are needed (not counting the Fallback SA)?
> If you rekey per-CPU SAs proactively you will always have a per-CPU SA ready.
> Or what do you mean by "race conditions" and why only few more SAs
> help in this case?

During rekey, you briefly have two SA's installed to prevent packet
leaks/fallback.  If both ends rekey at once, you might have two SA's
in flight plus the existing SA for a brief moment in time.

Also when one end has 2 CPUs and the other 4 CPUs, the end with 2 CPUs
will have more than one SA per CPU for the duration of the tunnel.

We tried to put the exact maximum in place in earlier drafts, accounting
for everything but during testing it just showed that this was not
workable for us.

>> Clearly, if for part of the connection, you are
>> using the fallback SA, you are running at suboptimal speed which is not
>> a situation you should remain in.
>
> Why? If the peer is unwilling (or unable) to install more SAs, then
> you will be in this suboptimal situation forever.

That is a different situation? Failure is always an option :)

I'm not sure I understand your point?

Using the fallback SA is well, a fallback that idealy won't trigger. But
in case something happened (eg a CPU got added to the running machine),
it is preferred to send packets out slowly over dropping them
completely. (on a highspeed link of 100gbps, you cannot cache all the
packets to cover 1 RTT for IKE to add the additional Child SA)

>  I don't think
> it's a wrong situation you want to escape from ASAP, I think it's
> a normal situation in general.

We disagree. Maybe we should call the fallback SA the emergency SA ? :)

>> Indeed. The idea is that no matter what, you can encrypt the packets and
>> send them, even if at sub-optimal speeds. We don't want packets to have
>> to wait another RTT for the per-CPU SA to establish. That would cause a
>> lot of issues (slow TCP retransmits, UDP application retransmits, etc).
>> Once the first SA is up, you have a working IPsec tunnel and no more
>> packets should be dropped or wait for SA's to establish.
>
> I understand all these considerations. My proposal is to use
> other existing per-CPU SA in this case. So, no special Fallback SA
> is needed (I understand that re-steering packet to a different CPU requires locking,
> but in my understanding using the Fallback SA requires locking as well).

To me that seems more complicated and issue prone, but I'll let Steffen
speak on this as an implementer.

>>>   The Fallback Child SA MUST NOT be deleted when idle, as
>>>   it is likely to be idle if enough per-CPU Child SAs are installed.
>>>
>>> I think that these BCP14 requirements make the fallback SA very special.
>>
>> Yes it does. It ensures there is a fully working (albeit slow) IPsec
>> connection.
>
> And the "specialityness" of this SA worries me. I think that the same
> functionality can be achieved without introducing this special SA.

How woud you guarantee that at least 1 per-CPU SA is always available to
be steered at by other CPUs ? What if that one is rekeying, how are you
going to sync that to all the other CPU configurations?

Paul