Re: [tcpm] CUBIC rfc8312bis / WGLC Issue 1 (Was: Re: ProceedingCUBIC draft - thoughts and late follow-up)

[Yoshifumi Nishida <[e-mail]>]

Hi Markku,

Thanks for the response. Yes, you got valid points. But, I still have some
comments.

First thing I would like to clarify is that we acknowledge the model used
for CUBIC has not been validated as you pointed out.
However, at the same time, I believe it doesn't mean the model has
significant threats to the Internet. We've never seen such evidence even
though CUBIC has been widely deployed for a long time. I am personally
thinking that we will need to see tangible evidence for the threats to
leave out the fact that it has been widely used.

The second thing I would like to mention is that I am not sure how many
drafts have been passed through the RFC5033 process.
For example, RFC8995, RFC9002 are congestion control related standard docs,
but in my understanding, the process had not been applied to them.
Some may say that because these proposals are not big threats, but from my
point of view, they are more aggressive than NewReno in some ways.
I am not sure what's the clear differences between CUBIC draft and them. I
personally haven't seen very solid evidence that they are not unfair to the
current standards.
We may need to redefine or enhance the process in the future, but at this
point, I personally don't have a strong reason to set a high bar only for
this draft. Because I believe all docs should be treated equally.
Hence, describing the fact that the CUBIC draft hasn't passed the RFC5033
process in the doc looks sufficient to me.

Thanks,
--
Yoshi


On Tue, Jun 14, 2022 at 8:02 AM Markku Kojo <[e-mail]> wrote:

> Hi Yoshi,
>
> I moved your comment and the discussion on your reply under this thread on
> the Issue 1 (see below)
>
> On Tue, 14 Jun 2022, Markku Kojo wrote:
>
> > Hi all,
> >
> > this thread starts the discussion on the issue 1: the incorrect model
> for
> > determining CUBIC alpha for the congestion avoidance (CA) phase (Issue 1
> a)
> > and the inadequate validation of a proper constant C for the CUBIC
> window
> > increase function (Issue 1 b).
> >
> >
> > Issue 1 a)
> > ----------
> >
> > The model that CUBIC uses to be fair to Reno CC (in Reno-friendly
> region) is
> > unvalidated and actually incorrect.
> >
> > A more detailed description of the issue:
> >
> > The original paper manuscript that CUBIC bases its behaviour in the
> > Reno-friendly region did a preliminary attempt to validate the model but
> > failed (and the paper never got published). This is the only known
> attempt to
> > validate the model and even this failed validation attempt was quite
> light,
> > consisting of only a couple of network settings and obviously did not
> use any
> > replications for the results shown in the paper. Hence, even the
> statistical
> > validity of the results remains questionable. Results were shown only
> for a
> > setting with AQM enabled at the bottleneck router. The results for a
> > tail-drop case are missing in the paper manuscript.
> >
> > The report (creno.pdf, see a pointer to the doc in the email pointed to
> > below) that Bob wrote provides some explanation why the model does not
> give
> > correct results and thereby the resulting behaviour presented in the
> original
> > paper notably deviates from that of Reno CC. The email that I wrote to
> the wg
> > list
> >
> > https://mailarchive.ietf.org/arch/msg/tcpm/bds-h_a6-NliTjx-ZqUSaFpSSnA/
> >
> > complements Bob's explanation for the AQM case and corrects Bob's
> analysis
> > for the tail-drop case, explaining why the model is incorrect for the
> > traditional and still today prevailing tail-drop router case.
> >
> > Consequently, the use of the incorect model results in unknown behaviour
> of
> > CUBIC when in the Reno-friendly region. Moreover, it is quite likely
> that the
> > behaviour is different with different AQM implementations at the
> bottleneck,
> > resulting in even more random behavior. This alone is very problematic
> and
> > becomes more problematic when considering how moving out from the
> > Reno-friendly region is specified: when the genuine CUBIC formula gives
> a
> > larger cwnd than the cwnd that the Reno-friendly model gives, CUBIC
> moves to
> > the genuine CUBIC mode that is significantly more aggressive than Reno
> CC.
> >
> > Therefore, if the incorrect model gives too low cwnd for mimicked Reno
> CC,
> > CUBIC moves too early to the genuine CUBIC mode and becomes too
> agggressive
> > too early even though it should behave equally aggressive as Reno CC. On
> the
> > other hand, if the incorrect model gives too large cwnd, CUBIC is too
> > aggressive throughout the Reno-friendly region.
> > In summary, if the model is not correct, it results in more aggressive
> > behaviour than Reno CC no matter which direction the model fails.
> >
> > And very importantly: some people have suggested that CUBIC should
> replace
> > the current stds track CC algos and become the default. The behaviour of
> Reno
> > CC is very thoroughly studied and very well understood. If we replace it
> with
> > *unknown* behaviour, how can we anymore specify what is the correct and
> > allowed aggressiveness for any upcoming CC when the behaviour of the new
> > default itself is unknown, making comparative analysis of other CCs
> against
> > CUBIC in the Reno-frindly region very difficult? The behaviour is
> assumed to
> > be the same as Reno CC but the actual behaviour is random, it may be 2
> times
> > or 8 times more aggressive than Reno, for example.
>
>
> On Tue, 7 Jun 2022, Yoshifumi Nishida wrote:
>
> > Hi Markku,
> >
> > Thanks for the detailed feedback. This is very useful.
> > One thing I would like to clarify is that we’ve already acknowledged the
> > TCP friendly  model in the draft has some unsolved discussions. But, I
> > believe our > current > consensus is to not change the logics for it in
> > the current draft as it will require long term evaluations.
> >
> > So, I would like to check if you’re suggesting we should update the
> > draft against it or you have some ideas to address these issues in
> > some ways (e.g adding more clarification in the draft, mentioning it in
> > the write-up, etc)
> >
> > Thanks,
> > --
> > Yoshi
>
> I think the problem is even trickier because it is hard to see how it
> would be possible to correct the model that is based on wrong
> assumptions. This said, it is important for the wg to consider whether it
> is ready to suggest publishing a congestion control algorithm that is not
> correct and has not been validated. And, if the answer is yes, how to
> justify it and what would be the appropriate status for the RFC as well
> as the way forward after publishing the draft.
>
> I fully symphatize those who have deployed CUBIC and understand that
> there is a pressure to publish the draft with no modifications to what has
> been implemented. However, RFC 5033 was written specifically to avoid this
> kind of situation where an CC algo has been (widely) deployed and only
> then brought to IETF standardization. It is understandable that those who
> have deployed the CC algo would be very reluctant to modify the algo. On
> the other hand, AFAIK all current stds track CC algos have had various
> issues that have been brought up during the standardization process but
> these issues have been resolved before publishing the draft. So, why we
> should make an exception? IMO, wide deployment cannot be the answer
> because it does not automatically reveal the negative impact to other
> traffic but specific comparative measurements must be carried out.
> Also, why should the IETF set a precedent for any future congestion
> control drafts, implying that it is ok to first deploy a CC algo and then
> bring it to IETF and use the (wide) deployment as an argument against
> modifying it regardless of whatever issues it might have?
>
> So, I don't have a good answer. IMO, if the draft is published with
> unresolved issues, the draft itself must clearly identify and document
> the issues and give some kind of justification and a clear way forward.
> That is, we must ensure there is an initiative set and path to follow in
> order to correct any shortcomings in a published RFC. Otherwise, the
> issues are very likely ignored and forgotten forever.
>
> Thanks,
>
> /Markku
>
>
>
> > Issue 1 b)
> > ----------
> >
> > Another issue related to the operating in the Reno-friendly region is
> the
> > question when CUBIC should operate in the Reno-friendly region and when
> it
> > may move out of it. Obviously CUBIC should stay in the Reno-friendly
> region
> > when Reno CC would be able to fully utilize the available network
> capacity.
> > In practice, this is specified by selecting the value for constant C in
> the
> > formula that is used to determine cwnd in the "genuine" CUBIC mode.
> However,
> > selecting a proper value for C has not been properly validated in a wide
> > range of environments as required in RFC 5033.
> >
> > Preliminary validation of constant C has been done for the original
> CUBIC
> > paper. That is good enough for a scientific paper but not adequate for
> an
> > IETF stds track algo. There seems to be no additional evaluation since
> the
> > timeframe of the CUBIC paper publication around 15 years ago.
> Particularly,
> > there seems to be no evaluation with AQM at the bottleneck router or
> with a
> > buffer-bloated bottleneck router, not to mention many other network
> > environments. Nor is there any data available for a non-SACK TCP sender.
> >
> > The evaluation of 1 a) and 1 b) must be done separately. Othserwise, it
> is
> > very hard to tell whether any deviations are due to the incorrect model
> or
> > incorrect value of C. The original CUBIC paper and some other papers
> show
> > that CUBIC is not fair to Reno CC in certain network conditions where
> Reno CC
> > has no problems in utilizing the available network capacity; instead,
> CUBIC
> > steals capacity from Reno CC.
> >
> > Thanks,
> >
> > /Markku
> >