Re: [IPsec] Discussion of draft-pwouters-ipsecme-multi-sa-performance

[Paul Wouters <[e-mail]>]

On Mon, 17 Oct 2022, Valery Smyslov wrote:

> implementation with say 10 CPUs. Does it make any difference for this implementation
> If it receives CPU_QUEUES with 100 or with 1000? It seems to me that in both cases
> it will follow its own local policy for limiting the number of per-CPU SAs,
> most probably capping it to 10.

That would be a mistake. You always want to allow a few more than the
CPUs you have. The maximum is mostly to protect against DoS attacks.
If you only have 10 CPUs, but the other end has 50, there shouldn't
be much issue to install 50 SA's. Not sure if we said so in the draft,
but you could even omit installing 40 of the outgoing SA's since you
would never be expected to use them anyway. but you have to install all
50 incoming ones because the peer might use them.

> Or do you want to say that the logic would
> be: well my peer has 1000 CPUs, it's not good for me to have more than 10,
> but let's be friendly and install 100, so that both of us are suffer...

At that point, it really also matters what the differences are between
the CPUs. An embedded device with 4 cpus at 800mhz vs a mega supercomputer
with 250 5ghz cpus. Already, counting CPUs is an approximation. Already,
the application needs proper threading to use multiple CPUs without a
single CPU bottleneck at the application layer. There might very well be
situations where multi-sa doesn't really help you at all.

But the simple use case is clear. I have a database cluster of 10 nodes
talking to eachother with lots of CPUs and most are idle because I have
one fully loaded CPU because my SA is bound to one CPU only.

> I don't think this logic is credible in real life, but even in this case
> there is already a mechanism that allows to limit the number of
> per-CPU SAs - it is the TS_MAX_QUEUE notify.

> So why we need CPU_QUEUES?

TS_MAX_QUEUE is conveying an irrecoverable error condition. It should
never happen. Where as CPU_QUEUES tells you how many per-CPU child SAs
you can do. This is meant to reduce the number of in-flight CREATE_CHILD_SA's
that will never become successful.

>>> I'm also not convinced that CPU_QUEUE_INFO is really needed, it mostly exists
>>> for debugging purposes (again if we get rid of Fallback SA). And I don't think we need
>>> a new error notify TS_MAX_QUEUE, I believe TS_UNACCEPTABLE can be used instead.

We did it to distinquish between "too many of the same child sa" versus
other errors in cases of multiple subnets / child SAs under the same IKE
peer. Rethinking it, I am no longer able to reproduce why we think it
was required :)

>>> there is no SA, this CPU requests IKE to create one for it. Meanwhile,
>>> the packet triggered this action can be 1) dropped 2) retained waiting for the SA to be ready
>>> or 3) re-steered to the CPU that already has an appropriate SA (it is indicated in the stub entry).
>>
>> On a preemptive system, the scheduler might migrate applications
>> from one cpu to another from time to time. So 1 and 2 are IMO not
>> appropriate as the application would stuck until a SA is created.
>> 3 has its own problems as discussed in the other mail.
>
> OK, but also see my considerations there.

The idea of the fallback SA is that you always have at least one child
SA guaranteed to be up that can encrypt and send a packet. It can be
installed to not be per-CPU. It's a guarantee that you will never need
to wait (and cache?) 1 RTT's time worth of packets, which can be a lot
of packets. You don't want dynamic resteering. Just have the fallback
SA "be ready" in case there is no per-cpu SA.

> I think it depends. I'd like to see optimization efforts to influence
> the protocol as less as possible. Ideally this should be local matter
> for implementations. This would allow them to interoperate
> with unsupporting implementations (and even to benefit from
> multi-SAs even in these situations).

Those that don't support this don't see notifies ? Or do you mean to
somehow install multiple SA's for the same thing on "unsupported"
systems? The problem currently is that when an identical child SA
is successfully negotiated, implementations differ on what they do.
Some allow this, some delete the older one. The goal of this draft
is to make the desire for multple idential child SAs very explicit.

Paul