Re: [bmwg] draft-lai-bmwg-*

["[e-mail]" <[e-mail]>]

Hi Vratko:

Thanks a lot for your comments!

As we are entering a new era of satellite Internet, it is foreseen that there will be a surge of new network protocols, functionalities, and devices tailored for emerging LEO satellite networks. It should be useful to evaluate them (from a network perspective) comprehensively in a controlled lab environment before launching them into space. This is the point that motivates us to consider a lab-level benchmarking methodology for network techniques/devices of LEO satellite networks.

I agree that we should focus more on a specific SUT/DUT (like a real protocol or device in satellite networks), and start with reasonably defined metrics to quantify the network performance. We spend much time discussing how to build a realistic satellite network simulator because we believe a hardware-in-the-loop simulation should be a viable path for benchmarking satellites in a laboratory environment.

In addition to RFC 2554, I also found that we can learn from RFC 5180 which provides some benchmarking guidelines to lead more complete and realistic evaluation for IPv6 networks.

Inventing more suitable metrics is also a very good suggestion : ) We will follow your comments to update our work accordingly.

Many thanks again!

Best,
Zeqi Lai


[e-mail]
 
From: bmwg-request
Date: 2023-07-27 05:10
To: bmwg
Subject: bmwg Digest, Vol 225, Issue 8
Send bmwg mailing list submissions to
[e-mail]
 
To subscribe or unsubscribe via the World Wide Web, visit
https://www.ietf.org/mailman/listinfo/bmwg
or, via email, send a message with subject or body 'help' to
[e-mail]
 
You can reach the person managing the list at
[e-mail]
 
When replying, please edit your Subject line so it is more specific
than "Re: Contents of bmwg digest..."
 
 
Today's Topics:
 
   1. draft-lai-bmwg-*
      (Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco))
 
 
----------------------------------------------------------------------
 
Message: 1
Date: Thu, 27 Jul 2023 12:10:48 +0000
From: "Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)"
<[e-mail]>
To: "[e-mail]" <[e-mail]>
Subject: [bmwg] draft-lai-bmwg-*
Message-ID:
<[e-mail]>
Content-Type: text/plain; charset="utf-8"
 
 
Hi, here are some assorted comments about the $subject drafts
(where * stands for istn-methodology-03 [0] and sic-benchmarking-02 [1]).
 
I do not see notification e-mails on bmwg list for those two drafts
when a new version is uploaded (like I see [2] for MLRsearch).
Is that because those two drafts are not adopted by BMWG?
If that is so, I encourage authors to notify the list manually.
(Applies to all drafts, not just the two this e-mail is about.)
 
I am not sure what is the difference in scope of those two drafts,
so I will treat them as one document for now.
 
There are multiple new ideas (compared to other BMWG RFCs and drafts).
Let me try to outline how I see the higher level new ideas:
1. New type of a big network (connectivity, bandwidth and latency are more dynamic).
2. The real deal is too big for a lab, we need to use a simulator.
3. The simulator should be realistic enough.
4. Which existing (or new) tests to run on the simulated SUT.
 
As Sarah mentioned, BMWG focuses on benchmarking in a lab.
Other working groups such as IPPM [3] may be able to measure
some performance related metrics on the real constellations directly.
 
Constellations are similar to the Internet as a whole,
in a sense that users are interested in end-to-end performance.
While you can test a single DUT, or a small SUT consisting
of few devices in a lab, the Internet end-to-end performance
also depends on link failures, routing stability and similar ?global? phenomena.
I expect you can similarly test a SUT consisting of only a few
(simulated) satellites, but the ?global? behavior
seems to be the focus of the drafts instead.
Do we have any documents for testing simulations of big
(up to Internet scale) dynamic networks?
 
While the question of creating ?realistic enough? simulations is important,
I am not sure it falls within BMWG scope.
Are there (I mean here in bmwg) any people trying to estimate
performance of a big datacenter by measuring various simulations of datacenters,
and trying to standardize parameters and realism of such simulations?
 
On the other hand, (simulations of) satellite constellations
are good examples of SUT consisting of many fallible elements,
and I personally think those will be more common in the future.
Such systems are not only nondeterministic (weather?),
they also may exhibit loss spikes in fairly predictable time intervals.
Current metrics (such as RFC 2544 throughput) have poor repeatability in such cases,
and we may want invent more suitable metrics.
 
Speaking about metrics, RFC 2544 throughput definition does contain
time constants (how long the trial traffic should last,
how long to wait for late frames), so realism of simulators
may come into scope after all, if they have configurable scaling of time.
E.g. path of a packet can be simulated faster
than speed of light allows in real life.
 
Vratko.
 
[0] https://datatracker.ietf.org/doc/draft-lai-bmwg-istn-methodology/
[1] https://datatracker.ietf.org/doc/draft-lai-bmwg-sic-benchmarking/
[2] https://mailarchive.ietf.org/arch/msg/bmwg/LhFsJQgZ_Ib-WSbHZKfRr6ynJ1A/
[3] https://datatracker.ietf.org/group/ippm/about/
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailarchive.ietf.org/arch/browse/bmwg/attachments/20230727/db221841/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 92 bytes
Desc: image001.gif
URL: <https://mailarchive.ietf.org/arch/browse/bmwg/attachments/20230727/db221841/attachment.gif>
 
------------------------------
 
Subject: Digest Footer
 
_______________________________________________
bmwg mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/bmwg
 
 
------------------------------
 
End of bmwg Digest, Vol 225, Issue 8
************************************