Re: [IPsec] Discussion of draft-pwouters-ipsecme-multi-sa-performance

[Valery Smyslov <[e-mail]>]

Hi Paul,

> On Tue, 18 Oct 2022, Valery Smyslov wrote:
> 
> >> I think this is implementation specific. You could install an temporary
> >> rule into the SPD that would give the fallback SA more priority than the
> >> per-CPU policy installed, so it wouldn't generate ACQUIRES for a while.
> >
> > Why for a while? And for how long? There is no indication
> > from the peer whether inability to create more SAs is temporary
> > or permanent, so we may end up with wasting resources when one peer will be constantly
> > trying to install per-CPU SA with no success.
> 
> That's why we propose the hint of NUM_QUEUES :)
> And why we have:
> 
>     The TS_MAX_QUEUE notify conveys that the peer is unwilling to create more
>     additional Child SAs for this particular Traffic Selector set.
> 
> Each peer knows the amount of this child SA they have. So if they
> receive a few Delete/Notifies perhaps than they can install more again
> if needed.

OK, but the same behavior can be implemented without NUM_QUEUES.

> >> Perhaps userland could also decide to terminate another per-CPU SA that
> >> is idle. Although I think the advised policy is stated to install at
> >> least one per-CPU SA per CPU (and allow a few more to catch any race
> >> conditions and rekeys).
> >
> > Why more SAs than the number of CPUs are needed (not counting the Fallback SA)?
> > If you rekey per-CPU SAs proactively you will always have a per-CPU SA ready.
> > Or what do you mean by "race conditions" and why only few more SAs
> > help in this case?
> 
> During rekey, you briefly have two SA's installed to prevent packet
> leaks/fallback.  If both ends rekey at once, you might have two SA's
> in flight plus the existing SA for a brief moment in time.

Yes, but they all are transient. I don't count them.

> Also when one end has 2 CPUs and the other 4 CPUs, the end with 2 CPUs
> will have more than one SA per CPU for the duration of the tunnel.

True, but this is not concerned with race conditions.
To put it more accurately - why do you need more SAs than
the largest of the number of CPUs for two peers (not counting the fallback SA
and any transient SAs that are created in the process of rekey)?

> We tried to put the exact maximum in place in earlier drafts, accounting
> for everything but during testing it just showed that this was not
> workable for us.

Still curious why it was not workable...

> >> Clearly, if for part of the connection, you are
> >> using the fallback SA, you are running at suboptimal speed which is not
> >> a situation you should remain in.
> >
> > Why? If the peer is unwilling (or unable) to install more SAs, then
> > you will be in this suboptimal situation forever.
> 
> That is a different situation? Failure is always an option :)
> 
> I'm not sure I understand your point?
> 
> Using the fallback SA is well, a fallback that idealy won't trigger. But
> in case something happened (eg a CPU got added to the running machine),
> it is preferred to send packets out slowly over dropping them
> completely. (on a highspeed link of 100gbps, you cannot cache all the
> packets to cover 1 RTT for IKE to add the additional Child SA)

You assume that the peers are able to handle the number of SAs,
that is equal to the largest of the number of CPUs they both have.
I describe a situation when the peer advertising fewer CPU cannot
handle that many SAs (e.g. it uses HSM and can only install limited 
number of SAs). In this situation this peer rejects creating additional
SAs with TS_MAX_QUEUE, so peers end up with smaller number of SAs.
This configuration is sub-optimal for the peer that have more CPUs,
but in my opinion it is a normal situation, not an error situation.

> >  I don't think
> > it's a wrong situation you want to escape from ASAP, I think it's
> > a normal situation in general.
> 
> We disagree. Maybe we should call the fallback SA the emergency SA ? :)

Will this improve the situation I described above ? :-)

> >> Indeed. The idea is that no matter what, you can encrypt the packets and
> >> send them, even if at sub-optimal speeds. We don't want packets to have
> >> to wait another RTT for the per-CPU SA to establish. That would cause a
> >> lot of issues (slow TCP retransmits, UDP application retransmits, etc).
> >> Once the first SA is up, you have a working IPsec tunnel and no more
> >> packets should be dropped or wait for SA's to establish.
> >
> > I understand all these considerations. My proposal is to use
> > other existing per-CPU SA in this case. So, no special Fallback SA
> > is needed (I understand that re-steering packet to a different CPU requires locking,
> > but in my understanding using the Fallback SA requires locking as well).
> 
> To me that seems more complicated and issue prone, but I'll let Steffen
> speak on this as an implementer.

OK.

> >>>   The Fallback Child SA MUST NOT be deleted when idle, as
> >>>   it is likely to be idle if enough per-CPU Child SAs are installed.
> >>>
> >>> I think that these BCP14 requirements make the fallback SA very special.
> >>
> >> Yes it does. It ensures there is a fully working (albeit slow) IPsec
> >> connection.
> >
> > And the "specialityness" of this SA worries me. I think that the same
> > functionality can be achieved without introducing this special SA.
> 
> How woud you guarantee that at least 1 per-CPU SA is always available to
> be steered at by other CPUs ? What if that one is rekeying, how are you
> going to sync that to all the other CPU configurations?

If the only remaining per-CPU Sa is deleted (e.g. due to lack of traffic),
it will be re-created when an outgoing packet appears, as with any usual SA. 
The syncing is costly, but happens infrequently.

Regards,
Valery.

> Paul