[bmwg] draft-lai-bmwg-*

["Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)" <[e-mail]>]

Hi, here are some assorted comments about the $subject drafts
(where * stands for istn-methodology-03 [0] and sic-benchmarking-02 [1]).

I do not see notification e-mails on bmwg list for those two drafts
when a new version is uploaded (like I see [2] for MLRsearch).
Is that because those two drafts are not adopted by BMWG?
If that is so, I encourage authors to notify the list manually.
(Applies to all drafts, not just the two this e-mail is about.)

I am not sure what is the difference in scope of those two drafts,
so I will treat them as one document for now.

There are multiple new ideas (compared to other BMWG RFCs and drafts).
Let me try to outline how I see the higher level new ideas:
1. New type of a big network (connectivity, bandwidth and latency are more dynamic).
2. The real deal is too big for a lab, we need to use a simulator.
3. The simulator should be realistic enough.
4. Which existing (or new) tests to run on the simulated SUT.

As Sarah mentioned, BMWG focuses on benchmarking in a lab.
Other working groups such as IPPM [3] may be able to measure
some performance related metrics on the real constellations directly.

Constellations are similar to the Internet as a whole,
in a sense that users are interested in end-to-end performance.
While you can test a single DUT, or a small SUT consisting
of few devices in a lab, the Internet end-to-end performance
also depends on link failures, routing stability and similar “global” phenomena.
I expect you can similarly test a SUT consisting of only a few
(simulated) satellites, but the “global” behavior
seems to be the focus of the drafts instead.
Do we have any documents for testing simulations of big
(up to Internet scale) dynamic networks?

While the question of creating “realistic enough” simulations is important,
I am not sure it falls within BMWG scope.
Are there (I mean here in bmwg) any people trying to estimate
performance of a big datacenter by measuring various simulations of datacenters,
and trying to standardize parameters and realism of such simulations?

On the other hand, (simulations of) satellite constellations
are good examples of SUT consisting of many fallible elements,
and I personally think those will be more common in the future.
Such systems are not only nondeterministic (weather?),
they also may exhibit loss spikes in fairly predictable time intervals.
Current metrics (such as RFC 2544 throughput) have poor repeatability in such cases,
and we may want invent more suitable metrics.

Speaking about metrics, RFC 2544 throughput definition does contain
time constants (how long the trial traffic should last,
how long to wait for late frames), so realism of simulators
may come into scope after all, if they have configurable scaling of time.
E.g. path of a packet can be simulated faster
than speed of light allows in real life.

Vratko.

[0] https://datatracker.ietf.org/doc/draft-lai-bmwg-istn-methodology/
[1] https://datatracker.ietf.org/doc/draft-lai-bmwg-sic-benchmarking/
[2] https://mailarchive.ietf.org/arch/msg/bmwg/LhFsJQgZ_Ib-WSbHZKfRr6ynJ1A/
[3] https://datatracker.ietf.org/group/ippm/about/