[hp-wan] Re: Notes from hp-wan informal virtual meeting, 9 Jan

[Dan Wing <[e-mail]>]

On Feb 22, 2025, at 7:24 AM, kehan yao <[e-mail]> wrote:
> 
> Hi Dan,
> 
> Thank you for your comments. Please see my reply inline below.
> 
> Best,
> Kehan
> 
> Dan Wing <[e-mail] <mailto:[e-mail]>> 于2025年2月21日周五 07:56写道：
>> On Feb 20, 2025, at 2:49 AM, kehan yao <[e-mail] <mailto:[e-mail]>> wrote:
>>> 
>>> Hi Dan,
>>> 
>>> Please see my previous email, we just submitted a new document on hpwan use cases and requirements--from public operator's view. Some of the requirements mentioned in the draft are related to H2N signalling. 
>>> 
>>> https://datatracker.ietf.org/doc/draft-yx-hpwan-uc-requirements-public-operator/
>>> 
>>> Please help review the document and welcome for comments.
>> 
>> Thanks.  
>> 
>> Section 3.1 describes transfer of 2.5TB of data can achieve only sustained 700Mbps over a 2000km path over a link that provides 7Gbps throughput.  Plugging numbers into a bandwidth delay calculator (e.g., https://www.speedguide.net/bdp.php) with 25ms delay (which seems appropriate for 2000km path), the TCP window size should be 21875000 bytes.  Is the receiver's window size at least that big?
>  
>> [KY] I just used the online calculation tool you provided. When I entered 700Mbps and 25ms, the result of RWIN is 2187500 bytes, so 2.19MB I think is reasonable in our case when the transfer tool is FTP/RSYNC. You said is 21875000 bytes, so maybe a typo?
>> 
>> Also, Since we use the Linux OS, it supports WINSCALE just as you mentioned in the questions below. The default value of TCP RWIN 65535 bytes can not guarantee the throughput requirements under high-BDP scenario. During our test, we tune the parameters in Linux OS to increase the RWIN of TCP to be over 20MB (sysctl -w net.core.wmem_max / sysctl -w net.core.wmem_default) so that end-to-end throughput can be guaranteed. We tested 64/128 connections for multi-stream transmission, and it take up about ~ x GB memory space so it won’t incur much overhead for the memory of the receiver side.  


Ok. Sounds like things were tuned to allow the bandwidth to be consumed.


>> What TCP congestion control algorithm is the sender using?
>> [KY]As for most of the FTP/RSYNC tools, the congestion control algorithm is CUBIC.  
>  
>> Are the sender and receiver both ECN-aware?  Does the entire network path support ECN?  Or its bottleneck links?  Or not at all?
>> [KY]In China Mobile’s backbone network, ECN is not supported at both endpoint and network side.
>  
>> Are there TCP Performance Enhancing Proxies (PEP) on the path?  (https://datatracker.ietf.org/doc/html/rfc3135)
>  
>> [KY]First, I admit that I’m not familiar with PEP as you mentioned and in our network, PEP is also not supported. When I searched for it, it seems an existing TCP proxy which can split TCP connections and bring some mechanisms to optimize end-to-end throughput, like ACK aggregations.

Yeah, they can do all sorts of TCP manipulations.  All proprietary.  


>> But as I searched, it also says that TLS make PEP can’t deal with encryption traffic,

Eh, if they're just manipulating TCP, a PEP will work fine with TLS.  If the PEP is additionally doing caching (e.g., HTTP caching), yes, the PEP won't be able to do its job with TLS-encrypted traffic.


>> which makes it limited for use. One of the requirements of this document is to design some of a new proxy function that can support protocol transform, for example, from QUIC to RDMA over QUIC/encrypted UDP. This is to guarantee both throughput performance, as well as data encryption and security. 


Yeah, I noticed that requirement.

Such a protocol transformation proxy harms deployment of new protocol features -- ossifying the protocols.  A careful design can reduce the ossification risk.


>  
>> I expect both TCP peers support SACK and WINSCALE (RFC1323), considering it's shipping in every typical OS for the last decade or two. 
>> [KY]Yes, as I mentioned above, in the systems we tested(Linux OS). They support SACK and WINSCALE.
>  
>> It would be helpful if Section 3.1 answered all those questions, and probably more questions that people smarter than me might have.
>> 
>> 
>> 
>> Section 3.2: I can't figure out the *network* use case of Section 3.2.  Lots of acronyms but the network doesn't it's iWARP and doesn't know iWARP's needs.  
>  
>> [KY]Sorry for not making section 3.2 very clear. Let me clarify a bit more.
>> 
>> Section 3.2 targets on cross-DC traffic, and especially traffic (Cross DC AI training) which use RDMA as its underlay transport. The case is not an case that have already been deployed, but a case that will be deployed in the near future.
>> 
>> - Why this case be included?
>> * Since end-to-end throughput need to be guaranteed, and also low CPU overhead at endpoint(requirements for RDMA)
>> * The traffic will be extended from intra-DC to cross-DC and over Internet, so it needs to be securely delivered.(requirements for RDMA over an encrypted transport)
>> * We are designing modified RoCEv2 to meet the requirements mentioned above, but RoCEv2 is not an Internet transport standard. But we need an Internet standard to meet these requirements. (So I mentioned iWARP over an encrypted transport) 
>> 
>> - How this relate to the proxy requirement?
>> * Since in section 3.1, FTP/RSYNC works over TCP, we also want QUIC traffic to be high throughput, as well as the case in section 3.2, so the proxy could be expected to transform TCP/QUIC to be RDMA-capable. 
>  
>> Also, Section 3.2 says "The iWARP protocol suite works over TCP and can provide security guarantee", but https://datatracker.ietf.org/doc/html/rfc5042#section-5.4 says TLS is inappropriate (in its Section 5.4.2) and suggests using RDDP over TCP over DTLS over UDP/IP (its Section 5.4.3) or IPsec (its Section 5.4.4) -- none of which are *iWARP* itself but rather encryption underneath it, which all use AES.  I can't tell from the text of Section 3.2 if the bandwidth and delay requirements can be achieved with AES (which is commonly used with DTLS and IPsec) with an AES-NI-capable chipset.   I suggest a change like:
>> 
>> OLD:
>> The iWARP protocol suite works over TCP and can provide security guarantee, 
>> NEW:
>> The iWARP protocol suite works over TCP and can provide security guarantee when run over DTLS or IPsec (see Section 5.4 of [RFC5042]), 
>> [KY]Thanks for the modification.
>  
>> and also add text that mentions DTLS and IPsec typically use AES encryption.  I quickly found https://medium.com/asecuritysite-when-bob-met-alice/whats-the-fastest-symmetric-cipher-and-mode-3d6e77841c2b that compares various AES modes and also includes comparison to SM4.  Are the sender or receiver capable of higher AES speeds than is seen over the network?
>> [KY]Sorry that I’m not very familiar with many details of encryption algorithms. I’ll check what we use in our system.

Ok.  My point is mostly that each CPU core has a limit of bytes-per-second it could encrypt.


>> I don't want to repeat everything I asked in Section 3.1, but considering iWARP is over TCP, all those same TCP questions apply.
>> 
>> 
>> Section 3.3 could be clearer if it clarified which of the following it is describing:
>> 
>>  (a) a single flow (5-tuple) traversing both a dedicated network and the Internet ("non-dedicated network"), or 
>>  (b) if the text in that section is discussing two separate flows, like one flow from University-A to University-B over a dedicated network and a completely separate flow from University-A to home-user-with-fast-Internet-connection.
>> 
>> It seems to be describing two separate flows, one flow across a dedicated network that is already configured to expect the flow, and a separate flow over the Interent which ... cannot provide the desired characteristics of the flow?  What are the desired characteristics for the Internet flow, is it just a desire for high bandwidth or something else?
>  
>> [KY]I think section 3.3 is also a case that we foresee to be deployed. For example, think about a small company (which use public operator’s network) that wants to ask for some astronomical data from a research institute (which use R&E network). The flow need to be transmitted just as the first case as you mentioned.

Ok.

>> When it traverses the two networks, the QoS policies may change, bandwidth and priority. Security policies may also change, like policing traffic that extends a pre-defined threshold.  


By "Security policies may also change, like policing traffic that extends a pre-defined threshold.", I don't follow.  Is that referring to a DoS attack, or DDoS attack?  Or referring to over-consumption of bandwidth per hour or per month?  

-d



>>  
>> -d
>> 
>> 
>> 
>>> 
>>> Best,
>>> Kehan
>>> 
>>> <[e-mail] <mailto:[e-mail]>> 于2025年2月20日周四 11:47写道：
>>>> Hi Dan, thanks for coming up for your comments as well as the use cases and requirements to be post here.
>>>> 
>>>> It seems to be a common perspective to have host & network signaling in place when it comes to sophisticated coordination between host and network. as far as hp-wan is concerned, the key issue to be addressed here is provide some mutual transparency between the host and network, to maximize the end2end throughput as well as completion time. 
>>>> 
>>>> So it's worthy diving deep into the use cases to identify the requirement commonality if it could be agreed it exists.
>>>> 
>>>> 
>>>> 
>>>> look forward to having further discussions here in the list, and also at IETF 122 next month.
>>>> 
>>>> 
>>>> 
>>>> Kind regards,
>>>> 
>>>> Daniel Huang
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Original
>>>> From: DanWing <[e-mail] <mailto:[e-mail]>>
>>>> To: Tim Chown <[e-mail] <mailto:[e-mail]>>;
>>>> Cc: 熊泉00091065;[e-mail] <mailto:[e-mail]> <[e-mail] <mailto:[e-mail]>>;Mohamed Boucadair <[e-mail] <mailto:[e-mail]>>;[e-mail] <mailto:[e-mail]> <[e-mail] <mailto:[e-mail]>>;[e-mail] <mailto:[e-mail]> <[e-mail] <mailto:[e-mail]>>;
>>>> Date: 2025年02月20日 08:09
>>>> Subject: [hp-wan] Re: Notes from hp-wan informal virtual meeting, 9 Jan
>>>> _______________________________________________
>>>> hp-wan mailing list -- [e-mail] <mailto:[e-mail]>
>>>> To unsubscribe send an email to [e-mail] <mailto:[e-mail]>
>>>> On Feb 19, 2025, at 2:17 AM, Tim Chown <[e-mail] <mailto:[e-mail]>> wrote:
>>>>> 
>>>>> Hi,
>>>>>  
>>>>> There has been quite a lot of previous work in the IETF in this area. Do you think the hp-wan signalling requirements that you’ve identified are capture din this draft?  If so, that’s great, if not then I’d assume the authors would welcome input both on requirements but also use cases.
>>>> 
>>>> Yep, send both requirements and use cases.
>>>> 
>>>>  
>>>>> The draft is focused on requirements, but an inventory of solutions would also be interesting to see. I presume the fact the authors are working on this draft indicates they don’t see any existing signalling mechanisms meet their requirements?
>>>> 
>>>> Right.  There is especially a lack of host-to-network signaling; it's heuristics and guessing intent.
>>>> 
>>>> more below.
>>>> 
>>>> 
>>>> 
>>>> 
>>>>>  
>>>>> Tim
>>>>>  
>>>>> On 19/02/2025, 03:22, "[e-mail] <mailto:[e-mail]>" <[e-mail] <mailto:[e-mail]>> wrote:
>>>>>  
>>>>>  
>>>>> Hi Zahed and Med,
>>>>> 
>>>>>  
>>>>> Sorry for the late reply~
>>>>> 
>>>>> Thank Med to bring the topic "Requirements for Host-to-Network Collaboration Signaling" to hp-wan.
>>>>> 
>>>>>  
>>>>> I read this I-D draft-kwbdgrr-tsvwg-net-collab-rqmts-04 and I think it has common host-to-network collaboration signalling requirement with hp-wan.
>>>>> 
>>>>> HP-WAN are discussing the use caces and related specific service requirement such as completion time, efficient use of capacity, high throughput which demands more collaboration between host and network. 
>>>>> 
>>>>> For example, the host in HP-WAN may provide traffic patterns to network and network may provide active network-collaborated scheduling and acknowledgement (such as quota or bandwidth scheduling, admission control and so on) to assure the negotiated rate of the traffic. 
>>>>> 
>>>>> During the signalling, per-packet metadata may be required in HP-WAN but need to discussed in details. 
>>>>> 
>>>> Per-packet metadata did not receive consensus interest in TSVWG.  But perhaps we were explaining it wrong, I don't know.
>>>> 
>>>>>  And I also agree with that "Host-to-Network Collaboration Signaling" is not in the scope of Scone WG. But I am not sure it is in scope of Tsvwg.
>>>>> 
>>>> 
>>>> 
>>>> Well, SCONE is doing host-to-network collaborative signaling.  SCONE has three requirements documents of varying scope (*), but none are WG documents and they don't have a requirements document in their charter (**).  
>>>> 
>>>> (*) https://datatracker.ietf.org/group/scone/documents/
>>>> (**) https://datatracker.ietf.org/group/scone/about/
>>>> 
>>>> -d
>>>> 
>>>> 
>>>>> 
>>>>>  
>>>>> Best Regards,
>>>>> 
>>>>> Quan
>>>>> 
>>>>>  
>>>>> Original
>>>>> From: ZaheduzzamanSarker <[e-mail] <mailto:[e-mail]>>
>>>>> To: [e-mail] <mailto:[e-mail]> <[e-mail] <mailto:[e-mail]>>;
>>>>> Cc: Tim Chown <Tim.Chown=[e-mail] <mailto:Tim.Chown=[e-mail]>>;[e-mail] <mailto:[e-mail]> <[e-mail] <mailto:[e-mail]>>;[e-mail] <mailto:[e-mail]> <[e-mail] <mailto:[e-mail]>>;
>>>>> Date: 2025年02月01日 04:13
>>>>> Subject: [hp-wan] Re: Notes from hp-wan informal virtual meeting, 9 Jan
>>>>> _______________________________________________
>>>>> hp-wan mailing list -- [e-mail] <mailto:[e-mail]>
>>>>> To unsubscribe send an email to [e-mail] <mailto:[e-mail]>
>>>>> Hi Med,
>>>>>  
>>>>> The placing of that particular mentioning scone is kind of off the contexr here. The discussion was about doing more informed rate adaptation , admission control where network load or bandwidth information can be used. There I mentioned such throughput guidance work is currently done in IETF at scone. It was not meant tobe scone itself will be used in hp-wan, it was just an example of network information sharing protocol work.
>>>>>  
>>>>> I hope it clarifies the context.
>>>>>  
>>>>> // Zahed
>>>>>  
>>>>> On Fri, 31 Jan 2025 at 14:40, <[e-mail] <mailto:[e-mail]>> wrote:
>>>>> Hi Zahed, all, 
>>>>>  
>>>>> I have one question about this comment:
>>>>>  
>>>>> ==
>>>>> Zahed: Applications can label and classify data packets, and then inform the network of their priority. The Scone working group seems to be dealing with such issues. Routing can be pre-configured to provide determinism - does the IETF currently have similar work?
>>>>> ==
>>>>>  
>>>>> I’m not sure to understand the SCONE mention here as that WG is only about signaling throughput from the network to a host. However, TSVWG used to discuss similar constructs to inform the network about the prioritit within ** THE SAME FLOW ** (Requirements for Host-to-Network Collaboration Signaling <https://datatracker.ietf.org/meeting/121/materials/slides-121-tsvwg-sessa-requirements-for-host-to-network-collaboration-signaling-00>) but seems there is no energy in that WG to work on these problems.
>>>>>  
>>>>> Cheers,
>>>>> Med
>>>>>  
>>>>> De : Tim Chown <Tim.Chown=[e-mail] <mailto:[e-mail]>>
>>>>> Envoyé : mercredi 29 janvier 2025 17:50
>>>>> À : [e-mail] <mailto:[e-mail]>
>>>>> Objet : [hp-wan] Notes from hp-wan informal virtual meeting, 9 Jan
>>>>>  
>>>>>  
>>>>> With thanks to Adrian and Quang.
>>>>>  
>>>>> Tim
>>>>>  
>>>>> ---
>>>>>  
>>>>> Minutes for HP-WAN online meeting (9th Jan 9:00~10:30 am GMT)
>>>>>  
>>>>> Attendance (23)
>>>>> Zahed, Tim Chown,Adrian,Daniel King,Daniel Huang,Quan Xiong,Yao Liu,Chumeng Wang,Xueyan Song,Kehan Yao,Zongpeng Du,Shiping Xu,Peng Liu,Cancan Huang,
>>>>> Huiyue Zhang,Zhengxin Han,Mengyao Han,Junfeng Zhao,Hang Shi,Antoine Fressancourt,Michael Welzl,David Millman,Christopher Walker
>>>>>  
>>>>> Notes captured with thanks to Adrian Farrel and Quan Xiong.
>>>>>  
>>>>> The main presentation to seed discussion at the meeting was given by Daniel Huang.
>>>>> See https://github.com/xiongquan1230/HP-WAN-Online-Meeting-Jan-9
>>>>>  
>>>>> Main discussion points include the following: 
>>>>>  
>>>>> Scope of work
>>>>> - The scope of the problem is narrowing (which seems to be a good thing).
>>>>> - Still a need to show the demarcation between WIT/Transport and Routing areas.
>>>>> - Requirements could be numbered, then the use cases can point at specific requirements. That would allow us to see where we need to focus.
>>>>>  
>>>>> State of Art
>>>>> - Need to update the state of the art draft with use cases, and more details (e.g., specific numbers for OTT traffic), and application examples (such as BW, file sizes, frequency, transmission and completion times, etc.)
>>>>> - Dan plans to chase Tim, Rodney (Arena PAC HPC) and Nick (ESNET) for examples
>>>>> - Need to focus on enabling technology discussion and requirements on WIT-related technologies
>>>>> - Work with Michael on reviewing his congestion control research and see if anything applies to current HP-WAN discussions
>>>>>  
>>>>> Should we have a WG-forming BoF in Bangkok?
>>>>> - General opinion is that we are nowhere near ready.
>>>>> - Bolstered by Zahed pointing out that we only get one more chance at an IETF BoF as there is a limit of two
>>>>> - Need a better understanding of the use cases, requirements, and to have a first stab at solutions work
>>>>>  
>>>>> Proposed next steps?
>>>>> - Use of Internet Drafts and the mailing list is essential
>>>>> - Issue new versions of the I-Ds to reflect discussions and firm up use cases /requirements
>>>>> - Encourage list discussion of the drafts
>>>>> - Plan for side meeting at IETF 122 with the intention of getting focus on what work the IETF might do
>>>>> - Discuss/Document early solution proposals, or at least applicable technology and what might need to be modified
>>>>> - Determine whether the problem/solutions are WIT area related (and thus pick an AD)
>>>>> - Plan for WG Forming BOF at IETF 123 *if* everything lines up
>>>>>  
>>>>> The following are more detailed questions and comments from attendees during the meeting:
>>>>>  
>>>>> Zahed: Not sure about the requirement of completion time, especially for the specific completion time. The range of requirements for completion time is large, for example, for 1TB data, there is a huge difference in the completion time requirements between 60 seconds and 60 minutes, so more specific application examples and accurate numbers need to be provided. Whether it requires deterministic behavior and the use of condition control mechanisms?
>>>>> What does multiple flows coexistence refer to? HP-WAN multiple flows, or competition with other flows?
>>>>>  
>>>>> Daniel: Completion time will be for small packets but not for bulk flows.
>>>>>  
>>>>> Kehan: From the perspective of China Mobile's operators, cross-DC Training is a promising application instance, although it has not been widely deployed yet, it has potential in the coming years.
>>>>>  
>>>>> Daniel K from chat: Good point Zahed. I think we have some good examples from Google Effingo, I'll take an action to capture some other examples from Janet (Tim) and ESNet (Nick).The average mentioned for Effingo is 1.4 EB <1s. We could include some specific application examples in the State-of-the-art I-D.Agree, more specific.
>>>>>  
>>>>> Quan Xiong from chat: I think multiple flows could be HP-WAN flows within an application or different applications. We also need to consider the flows concurrent in the WAN competing resources with HP-WAN.
>>>>>  
>>>>> Cancan Huang from chat: It has a prerequisite that the network is congested. It think in the massive data transmission scenario, congestion should avoid in advance. Since, compared to the small volume traffic, the packet retransmission in ultra large volume will lead to Network paralysis. So, the network resource such as link and bandwidth should be scheduled for specific flows in advance to avoid congestion.
>>>>>  
>>>>> Michael Welzl from chat:
>>>>> Some material: a Ph.D. thesis on the topic: 
>>>>> https://folk.universitetetioslo.no/michawe/research/students/kashif_minur-diss_final.pdf
>>>>> - combining admission and congestion control for minimal completion time. 
>>>>> This was perhaps the final paper from the above thesis, IIRC: 
>>>>> https://www.sciencedirect.com/science/article/abs/pii/S0167739X11002111?via%3Dihub
>>>>> This was work done in my first EC project, another group in our project that worked heavily on admission control + congestion control for FCT minimization was the group of Pascale Vicat-Blanc Primet in Lyon; I think she created a startup out of this work at some point, Google gave me this old page related to this work: 
>>>>> https://www.ens-lyon.fr/LIP/RESO/Software/NXE/
>>>>>  
>>>>> Michael Welzl from chat:
>>>>> I think scheduling and admission control can be done more opportunistically - without needing end host - network signaling, just based on how the congestion control algorithms play out. E.g.: I see that these 10 flows are going to be too slow, so I need to delay the 2 least urgent ones, and I won't admit a new one because I see that the FCT deadline is at risk.
>>>>>  
>>>>> Tim: There is no problem in lightly loaded networks, but some congestion is evident in limited bandwidth and some large traffic scenarios. There is in R&E space the ESnet work, seehttps://sense.es.net <https://sense.es.net/>, but deployment is only in specific limited networks to date.
>>>>>  
>>>>> Zahed: Avoiding congestion through signaling in the network is a common topic discussed by IETF, and there may already be solutions, but it is reasonable to propose this requirement in the HP-WAN scenario. Moreover, it involves the collaboration between the transport layer and routing layer, as well as  cross layer interfaces, requiring a clear distinction between the WIT/Transport and Routing.
>>>>>  
>>>>> Adrian: Agreed, there are already many signaling mechanisms in the IETF, including various mechanisms from the application layer to the network layer, but this faces challenges such as privacy, and the issue of flow priority also needs to be considered.
>>>>>  
>>>>> Zahed: Applications can label and classify data packets, and then inform the network of their priority. The Scone working group seems to be dealing with such issues. Routing can be pre-configured to provide determinism - does the IETF currently have similar work?
>>>>>  
>>>>> Adrian: There is no such work, but RTG has proposals, and OPS and WIT may also have limited exploration, but they are not yet mature.
>>>>>  
>>>>> Zahed: This is the WIT area and should focus on the technology and protocols of the transport layer. It is necessary to clearly distinguish the boundary between WIT/transport layer and RTG/routing. There may be potential impacts between coordination and scheduling, which may require technologies similar  to SDN controllers.
>>>>>  
>>>>> Tim: RFC 5434 specifies how to successfully host a BoF and the subsequent work should be carried out in accordance with its requirements, given we may want to hold that second and final (WG forming) BoF at some point.
>>>>>  
>>>>> Zahed: We should continue to drive discussions through email lists and updated drafts, and combine specific use cases and requirements to promote the formation of technical solutions. I suggest to hold a side meeting at IETF 122 (Bangkok) for focus discussion, and plan to initiate the WG BoF in the future.
>>>>>  
>>>>> ** End of meeting
>>>>> ____________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.  This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
>>>>> _______________________________________________
>>>>> hp-wan mailing list -- [e-mail] <mailto:[e-mail]>
>>>>> To unsubscribe send an email to [e-mail] <mailto:[e-mail]>
>>>> 
>>>> _______________________________________________
>>>> hp-wan mailing list -- [e-mail] <mailto:[e-mail]>
>>>> To unsubscribe send an email to [e-mail] <mailto:[e-mail]>
>>