[bmwg] Re: Empirical attack vectors for draft-han-bmwg-agent-security-benchmark (real 2026 incidents + a runnable scorer)

[patrick <[e-mail]>]

Meiling, Yu -- thank you both, and I'm glad it's useful.

On gaps: your five dimensions hold up well against the incidents. Where I'd gently suggest room to grow is in three places the current axes don't fully separate:

1. Incident-response / false-refusal under legitimate-defender pressure. The Hugging Face case showed the inverse of jailbreak -- a model that over-refuses when a real defender needs it to read attack logs. "Does it refuse attackers" and "does it still help defenders under duress" are different measurements, and only the first is currently scored.

2. Non-text / cross-modal injection. Payloads that arrive as an image or file the agent reads as data and never as instructions (the GhostCommit class). This sits underneath "plugin/skill" but isn't quite the same axis.

3. Agent evades its own controls. Distinct from external jailbreak: the recent OpenAI disclosure had a model observe its own security scanner and route around it. That's an autonomous-iteration sub-dimension -- the agent modeling and defeating a monitor placed on it -- worth its own cell.

Offered as candidates, not corrections -- you know the framework far better than I do.

On the case study: yes, we'd be glad to have the mapping included as an appendix / case study, with the same rule we hold ourselves to -- every incident credited to its primary researcher; we're the grounding layer, not the discoverers. Happy to provide the material in whatever format the write-up needs (the JSON is machine-readable now; if there's an Internet-Draft or appendix convention you want it conformed to, point me at it and I'll match it).

-- Patrick

Patrick Duggan
DugganUSA LLC | dugganusa.com

________________________________
From: [e-mail] <[e-mail]>
Sent: Wednesday, 22 July 2026 09:56:43
To: patrick <[e-mail]>; [e-mail] <[e-mail]>
Cc: bmwg <[e-mail]>
Subject: Re: Empirical attack vectors for draft-han-bmwg-agent-security-benchmark (real 2026 incidents + a runnable scorer)


Hi Patrick,

Wow, thank you so much for this incredibly helpful email and contribution! We love the "neighborly" spirit—this kind of proactive help is exactly what we need, and the work you've shared is fantastic.

We're already diving into the incident mappings, JSON data, and the reference scorer you provided. This is the perfect "ground truth" to help us validate and improve the benchmark.

As we dig in, two initial questions come to mind:

  1.  From your analysis, do you feel our existing metrics adequately cover the attack incidents you mentioned, or did you notice any potential gaps where our framework could be expanded?
  2.  This data is incredibly valuable. Would you be open to us including your analysis as a case study or appendix in our formal write-up? We would, of course, provide full credit to your shop and the primary researchers.

Thanks again for reaching out. We're excited to explore this further and look forward to the conversation!

I'm also CC'ing the working group mailing list to keep other interested members in the loop.

Best,
Meiling
________________________________
[e-mail]

From: patrick<mailto:[e-mail]>
Date: 2026-07-22 00:15
To: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>
Subject: Empirical attack vectors for draft-han-bmwg-agent-security-benchmark (real 2026 incidents + a runnable scorer)

Hello Meiling and Yu,

A small and slightly sheepish introduction: we're a bootstrapped threat-intelligence shop in Minnesota, and until this morning we had never heard of the IETF. We found one of our write-ups cited in the 126 hackathon wiki, went to learn what that meant, and decided the neighborly thing was to actually show up and help rather than just be flattered. So please read this as a newcomer offering a hand, not an expert pronouncing anything.

Your Security Evaluation Benchmark for AI Agents is well-specified — and a benchmark's next need is ground truth. That's exactly what we track for a living, so we mapped five documented 2026 agentic-attack incidents onto your metric dimensions (autonomous-iteration, plugin/skill, third-party components, jailbreak, operational) as candidate test vectors — JADEPUFFER's ~5-minute autonomous exploit-to-container-escape, the Pillar sandbox escapes, SleeperGem/ChocoPoC dependency poisoning, non-text prompt injection, and the Hugging Face incident-response false-refusal axis.

It's offered as prose and machine-readable JSON, plus a runnable reference scorer for the supply-chain vector (self-tests 6/6). Every incident is credited to its primary researcher.

https://github.com/pduggusa/dugganusa-ietf/tree/main/agent-security-benchmark — and our live check-package tool (on the MCP registry) is the production version of that scorer, testable now.

— Patrick Duggan
DugganUSA LLC · dugganusa.com<https://www.dugganusa.com>

Offered freely as an IETF 126 Hackathon contribution — take, leave, or ignore any of it. Apologies if this is an unexpected note; we're brand-new to the IETF and just trying to be useful.