[nmrg] draft-liu-nmrg-ai-llm-inference-requirements

[Libin Liu <[e-mail]>]

Hi Authors and WG，

I am interested in using LLM as the reasoning engine for various services and found this draft. I read it and have some questions/comments, and hope they could help.

1. Will this document only consider decoding-only LLMs? I am not sure if I misunderstood Fig. 1.
2. In Section 3, there are two suggested metrics (i.e., TTFT and TBT), which are used to evaluate the goodput performance of LLM service. I am curious why not use TPS (tokens per second) directly.
3. “Optimization aims to minimize both TTFT and TBT under resource constraints and SLO constraints.” What are the suggested SLO constraints?
4. In Section 4, I am not sure whether it is appropriate for the section title calls "network and system requirements", neither the document title. Sections 4.1, 4.2, and 4.3 all state the requirement for scheduling and management of the LLM service framework itself. The draft also said "frameworks MUST address the following ...". Are network and system requirements the requirements for low-level systems and networks (e.g., flops, memory, and bandwidth) that support the LLM services? Or requirements for the frameworks/scheduling/orchestrators that manage/schedule/run LLMs?
5. In Section 4, "MUST" restricts the working items for service optimization. Clearly, the factors which affects the proposed metrics (TTFT and TBT) are not limited to the three, more factors should be considered, such as LLM archs, hardware, resource scheduling algos, and prompts (workloads).

Please correct me if I am wrong.



Best,
Libin