[irtf-discuss] Networking for AI side meeting TOMORROW from 12:00 to 13:45 in Park Suite 4

[Antoine FRESSANCOURT <[e-mail]>]

Dear I {E, R} TFers,

We are going to hold a side meeting about Networking for AI tomorrow in Park Suite 4 from 12:00 to 13:45.

Our idea with this side meeting is to continue the discussions we initiated during previous IETF meetings about research directions on networking for AI that should happen in the IRTF, as a complement of the activities ongoing in the IETF.

The meeting will be introduced by a technical talk about benchmarks for Network Agents that will be of interest for both researchers and people deploying agents for network management.

Looking forward to see you tomorrow,

Antoine Fressancourt and Dirk Kutscher

--

https://github.com/IRTF-net4AI/IETF126-sidemeeting/

Logistics:
Room: Park Suite 4
Remote participation: https://ietf.webex.com/meet/sidemeetings1 (IETF Webex)

Agenda:

  *   12:00 - 12:05 (5 min) Meeting setup and introduction of the meeting
  *   12:05 - 12:35 (30 min) Gym & Benchmarks for Network Agents by Dario Rossi (Huawei)
  *   12:35 - 13:45 (70 min) Discussion on charter for future Research Group

Gym & Benchmarks for Network Agents (abstract)
This talk emphasizes that, to increase generalization capabilities of an agent in an automated fashion, it is necessary to minimize human intervention in the agentic evolution loop: Rather, human effort is needed to define benchmarks and evaluations practices. IETF is the right forum for vendors, ISPs, OTT, academia & (power) users to discuss and agree on useful realistic benchmarks.

Additionally, instead of showing the shiny aspects of agents, we prefer to show the dark side of their evaluation, i.e., when things can go wrong and benchmaxxing creates illusion of progress, or silent error and oddities plague the agent execution. For all these cases, we argue that, before classic accuracy metrics, community and benchmarks should explicitly track, count and expose failure modes to increase trust and ultimately foster adoption. The talk concludes with a bitter lesson (computational progress outpace human design), having nevertheless a sweet taste (open weight models progress is exponential, so that on-prem agents are a near future possibility).