Re: [Lsr] New Version Notification for draft-xu-lsr-flooding-reduction-in-clos-01.txt

[Vasilenko Eduard <[e-mail]>]

Hi Jeff,
5 topology hops is 5/3->66% worse than 3 hops for latency, reliability, and cost.
Why do you assume 5 hops if the needed scale is possible to achieve by 3 hops?
In fact, modern 1-ASIC switches (4.8Tbps, 51.2Tbps) permit 390k 100GE end-points for Megafly 3-hops topology (with 50% oversubscription) or 195k*100GE wire-speed.
Warning: Megafly in general demands equal load distribution at the application level – this restriction always exists for full mesh instead of centralized boxes.
PS: You said “stage” which probably means that the Server/Processor port is not included in the hops calculation. 3 topology hops with end-points are 5 hops overall.
Eduard
From: Lsr [mailto:[e-mail]] On Behalf Of Jeff Tantsura
Sent: Tuesday, November 28, 2023 2:32 AM
To: Robert Raszuk <[e-mail]>
Cc: Acee Lindem <[e-mail]>; Les Ginsberg (ginsberg) <ginsberg=[e-mail]>; Tony Li <[e-mail]>; [e-mail]; [e-mail]
Subject: Re: [Lsr] New Version Notification for draft-xu-lsr-flooding-reduction-in-clos-01.txt

Robert,

In context of LLM (10% of that for DLRM) training clusters, towards 2024/25 we would be looking to up to 8K end-points in a 3 stage leaf-spine fabric and up to 64-256K in 5 stages.
Virtualization and how it is instantiated might significantly change amount/distribution of state in underlay/overlay.

Obviously, these are hyperscale size deployments, many will be running 10-30 switches fabrics, where anything could work.
BGP seems to work just fine, some data plane signaling could be used as a near real time augmentation to “slow but stable” control plane.

Cheers,
Jeff


On Nov 26, 2023, at 14:30, Robert Raszuk <[e-mail]<mailto:[e-mail]>> wrote:

Hey Jeff,

Could you be so kind and defined term: "scaled-out leaf-spine fabrics" ?

Specifically folks watching us here would highly appreciate if we state max target nodes with vanilla ISIS and max target nodes when your ISIS implementation supports draft-ietf-lsr-dynamic-flooding<https://datatracker.ietf.org/doc/html/draft-ietf-lsr-dynamic-flooding>

While I am a BGP person I feel pretty strongly that BGP is not a best fit for the vast majority of DC fabrics in use today.

Cheers,
Robert


On Sun, Nov 26, 2023 at 10:49 PM Jeff Tantsura <[e-mail]<mailto:[e-mail]>> wrote:
I agree with all aforementioned comments.

Wrt AI/ML networking - if a controller is used, what is required is link state exposure northbound and not link state protocol  in the fabric. (I could argue for RIFT though ;-))
I’d urge you to take a look at Meta’s deployment  in their ML clusters (publicly available) - they use BGP as the routing protocol to exchange reachability (and build ECMP sets) and provide a backup if controller computed next hop goes away/before new one has been computed.
Open R is used northbound to expose the topology (in exactly same way - BGP-LS could be used).

To summarize: an LS protocol brings no additional value in scaled-out leaf-spine fabrics, without significant modifications -  it doesn’t work in irregular topologies such as DF, etc.
Existing proposals - there are shipping implementations and experience in operating it, have proven their relative value in suitable deployments.

Cheers,
Jeff

> On Nov 26, 2023, at 12:20, Acee Lindem <[e-mail]<mailto:[e-mail]>> wrote:
>
> Speaking as WG member:
>
> I agree. The whole Data Center IGP flooding discussion went on years ago and the simplistic enhancement proposed in the subject draft is neither relevant or useful now.
>
> Thanks,
> Acee
>
>> On Nov 24, 2023, at 11:55 PM, Les Ginsberg (ginsberg) <ginsberg=[e-mail]<mailto:[e-mail]>> wrote:
>>
>> Xiaohu –
>> I also point out that there are at least two existing drafts which specifically address IS-IS flooding reduction in CLOS networks and do so in greater detail and with more robustness than what is in your draft:
>> https://datatracker.ietf.org/doc/draft-ietf-lsr-distoptflood/
>> https://datatracker.ietf.org/doc/draft-ietf-lsr-isis-spine-leaf-ext/
>> I do not see a need for yet another draft specifically aimed at CLOS networks.
>> Note that work on draft-ietf-lsr-isis-spine-leaf-ext was suspended due to lack of interest in deploying an IGP solution in CLOS networks.
>> You are suggesting in draft-xu-lsr-fare that AI is going to change this. Well, maybe, but if so I think we should return to the solutions already available and prioritize work on them.
>>    Les
>>  From: Lsr <[e-mail]<mailto:[e-mail]>> On Behalf Of Tony Li
>> Sent: Thursday, November 23, 2023 8:39 AM
>> To: [e-mail]<mailto:[e-mail]>
>> Cc: [e-mail]<mailto:[e-mail]>
>> Subject: Re: [Lsr] New Version Notification for draft-xu-lsr-flooding-reduction-in-clos-01.txt
>> Hi,
>> What you’re proposing is already described in IS-IS Mesh Groups (https://www.rfc-editor.org/rfc/rfc2973.html) and improved upon in Dynamic Flooding (https://datatracker.ietf.org/doc/html/draft-ietf-lsr-dynamic-flooding).
>> Regards,
>> Tony
>>
>>
>> On Nov 23, 2023, at 8:29 AM, [e-mail]<mailto:[e-mail]> wrote:
>> Hi all,
>> Any comments or suggestions are welcome.
>> Best regards,
>> Xiaohu
>> 发件人: [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>
>> 日期: 星期三, 2023年11月22日 11:37
>> 收件人: Xiaohu Xu <[e-mail]<mailto:[e-mail]>>
>> 主题: New Version Notification for draft-xu-lsr-flooding-reduction-in-clos-01.txt
>> A new version of Internet-Draft draft-xu-lsr-flooding-reduction-in-clos-01.txt
>> has been successfully submitted by Xiaohu Xu and posted to the
>> IETF repository.
>>
>> Name:     draft-xu-lsr-flooding-reduction-in-clos
>> Revision: 01
>> Title:    Flooding Reduction in CLOS Networks
>> Date:     2023-11-22
>> Group:    Individual Submission
>> Pages:    6
>> URL:      https://www.ietf.org/archive/id/draft-xu-lsr-flooding-reduction-in-clos-01.txt
>> Status:   https://datatracker.ietf.org/doc/draft-xu-lsr-flooding-reduction-in-clos/
>> HTMLized: https://datatracker.ietf.org/doc/html/draft-xu-lsr-flooding-reduction-in-clos
>> Diff:     https://author-tools.ietf.org/iddiff?url2=draft-xu-lsr-flooding-reduction-in-clos-01
>>
>> Abstract:
>>
>>   In a CLOS topology, an OSPF (or ISIS) router may receive identical
>>   copies of an LSA (or LSP) from multiple OSPF (or ISIS) neighbors.
>>   Moreover, two OSPF (or ISIS) neighbors may exchange the same LSA (or
>>   LSP) simultaneously.  This results in unnecessary flooding of link-
>>   state information, which wastes the precious resources of OSPF (or
>>   ISIS) routers.  Therefore, this document proposes extensions to OSPF
>>   (or ISIS) to reduce this flooding within CLOS networks.  The
>>   reduction of OSPF (or ISIS) flooding is highly beneficial for
>>   improving the scalability of CLOS networks.
>>
>>
>>
>> The IETF Secretariat
>>
>> _______________________________________________
>> Lsr mailing list
>> [e-mail]<mailto:[e-mail]>
>> https://www.ietf.org/mailman/listinfo/lsr
>> _______________________________________________
>> Lsr mailing list
>> [e-mail]<mailto:[e-mail]>
>> https://www.ietf.org/mailman/listinfo/lsr
>
>
> _______________________________________________
> Lsr mailing list
> [e-mail]<mailto:[e-mail]>
> https://www.ietf.org/mailman/listinfo/lsr

_______________________________________________
Lsr mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/lsr