Re: [Tsv-art] [spring] TSV-ART review of draft-ietf-spring-segment-routing-msdc-08

[Mirja Kühlewind <[e-mail]>]

Hi Gaurav,

please see inline.

On 03.08.2018 07:20, Gaurav Dawra wrote:
> Hey Mirja,
>
> Sorry for the long delay. I was traveling constantly since IETF and 
> bit lost in my mailbox and discussion with Authors. Please see my 
> response inline[Gaurav]
>
> _
> I think with your changes you addressed explicit problems Martin 
> called out, however, I still have high level concerns about these 
> sections as they are mostly giving speculative recommendation which 
> are not clear to me to work in practice.
>
> Regarding section 7.1, you say _
> "A flowlet is defined as a burst of packets from the same flow 
> followed by an idle interval."
> _but then you say _
> "...then the application can break the elephant flow F into flowlets 
> F1, F2, F3, F4..."
>
> This sounds like you would just divide an elephant flow mostly 
> randomly into flowlets which can interact badly with the congestion 
> control. If you actually have chunks of data that are transmitted with 
> large enough idle interval in between (as you define flowlets in the 
> first sentence), that is not an elephant flow anymore and it will not 
> help you to "spread the load of the elephant flow through all the ECMP 
> paths". In summary I actually don't see how the concept of flowlets 
> can be helpful to address the problem you have at all, and I still 
> advise you to remove section 7.1 entirely.
>
> _[Gaurav] Hi Mirja, Thanks for the review. _The proposal here is no 
> different that current ECMP hashing at flowlet level. The only 
> difference which is being pointed out here is that if we use SR, we 
> could leverage on the ability of be aware of multiple disjoint paths 
> rather than the hashing. It’s pins the flowlets to particular paths 
> which is basic SR operations.
>

Yes the problem is that we usually don't recommend ECMP hashing on 
flowlet level because it can interact badly with the end-end mechanisms 
of that flow. As I said, I don't see how the notion of flowlets help you 
problem and therefore I still recommend to remove that paragraph.

> _
> Regarding section 7.2, I also still skeptical about any benefits that 
> can be achieved. Given you are in a data center, the controller should 
> already know about static parameters such as the maximum bandwidth per 
> link.
>
> _For dynamic parameters, e.g. like loss rate, measuring them on a 
> per-flow bases is the wrong thing to do. What I mean is you can ask a 
> router about the average loss rate that it observes and that might 
> give you some valuable, however, if you ask a TCP flow about the 
> average loss rate the answer will mainly depend on the congestion 
> controller used and the currently available bandwidth, which is a very 
> dynamic property and not a link characteristic. So this information is 
> not usable for performance aware routing. A flow could give you 
> information about the observed RTT (min/max) on a certain path, or the 
> maximum available bandwidth on a path, but as I said, given you are in 
> a data center environment these are information that the controller 
> already should have anyway.
>
> _[Gaurav] They are two separate mechanisms. _Most DCs have some sort 
> of data-plane/ECMP aware tracing mechanism to detect the loss/delays 
> and can be combined with Application back-off to detect issue. All 
> this section is suggesting is that SR can be used to pin the path to 
> particular set of ECMP paths instead of relying on ECMP hashing.
>

_This is not quite what the text says. If that is the statement you want 
to make, that is fine but then you don't need to talk about performance 
aware routing at all.
>
> _
> Your example with detecting a faulty path due to losses does not work 
> with TCP as you never know if these loses are due to a problem on the 
> path, self-induced or by a competing flow. And even if you don't use 
> TCP and e.g. send constant bit rate traffic, there may be a large 
> number of competing TCP flows that can induce the loses. Try to steer 
> traffic "away" on a time-scale that is slower than TCP dynamics or 
> even your flow dynamic (when flows start or end) in case you have a 
> lot of very short flow, in the best case doesn't work and in the worst 
> case leads to oscillation.
>
> _[Gaurav] As _I said above, there are other mechanisms to detect loss 
> and trace the path on which loss is seen. This is a common mechanism 
> used in MSDCs.
>
_I think this is beyond the scope of the document.
>
> _
>
> _
>
> _I am happy to discuss further over the phone to try to explain the 
> thought process. I will also do check again with Authors to update the 
> text or something else based on our conversation.
>

Maybe see if some update can be made to the text first and then we can 
have another discussion if needed.

> _
>
> _
>
> _Cheers,
>
> _
>
> _Gaurav
>
> If you want to make TCP use for handover situation where one path 
> might go away or become unusable, it's best to use Multipath TCP (with 
> coupled congestion control) instead because that works on the right 
> time scale. Again, I don't think this is a use case for SR and I would 
> recommend to remove the section entirely.
>
> Mirja _
>
> _
>
>
>
> On Mon, Jul 16, 2018 at 11:08 PM, Gaurav Dawra <[e-mail] 
> <mailto:[e-mail]>> wrote:
>
>     Hi Mirja,
>
>     Ack. Let me work with authors to close ASAP.
>
>     Cheers,
>
>     Gaurav
>
>     Sent from my iPhone
>
>     On Jul 5, 2018, at 10:06 AM, Mirja Kühlewind <[e-mail]
>     <mailto:[e-mail]>> wrote:
>
>>     Hi Gaurav,
>>
>>     sorry for my very long delay but this got somehow a bit lost in
>>     my mailbox.
>>
>>     I think with your changes you addressed explicit problems Martin
>>     called out, however, I still have high level concerns about these
>>     sections as they are mostly giving speculative recommendation
>>     which are not clear to me to work in practice.
>>
>>     Regarding section 7.1, you say
>>     "A flowlet is defined as a burst of packets from the same flow
>>     followed by an idle interval."
>>     but then you say
>>     "...then the application can break the elephant flow F into
>>     flowlets F1, F2, F3, F4..."
>>
>>     This sounds like you would just divide an elephant flow mostly
>>     randomly into flowlets which can interact badly with the
>>     congestion control. If you actually have chunks of data that are
>>     transmitted with large enough idle interval in between (as you
>>     define flowlets in the first sentence), that is not an elephant
>>     flow anymore and it will not help you to "spread the load of the
>>     elephant flow through all the ECMP paths". In summary I actually
>>     don't see how the concept of flowlets can be helpful to address
>>     the problem you have at all, and I still advise you to remove
>>     section 7.1 entirely.
>>
>>     Regarding section 7.2, I also still skeptical about any benefits
>>     that can be achieved. Given you are in a data center, the
>>     controller should already know about static parameters such as
>>     the maximum bandwidth per link. For dynamic parameters, e.g. like
>>     loss rate, measuring them on a per-flow bases is the wrong thing
>>     to do. What I mean is you can ask a router about the average loss
>>     rate that it observes and that might give you some valuable,
>>     however, if you ask a TCP flow about the average loss rate the
>>     answer will mainly depend on the congestion controller used and
>>     the currently available bandwidth, which is a very dynamic
>>     property and not a link characteristic. So this information is
>>     not usable for performance aware routing. A flow could give you
>>     information about the observed RTT (min/max) on a certain path,
>>     or the maximum available bandwidth on a path, but as I said,
>>     given you are in a data center environment these are information
>>     that the controller already should have anyway.
>>
>>     Your example with detecting a faulty path due to losses does not
>>     work with TCP as you never know if these loses are due to a
>>     problem on the path, self-induced or by a competing flow. And
>>     even if you don't use TCP and e.g. send constant bit rate
>>     traffic, there may be a large number of competing TCP flows that
>>     can induce the loses. Try to steer traffic "away" on a time-scale
>>     that is slower than TCP dynamics or even your flow dynamic (when
>>     flows start or end) in case you have a lot of very short flow, in
>>     the best case doesn't work and in the worst case leads to
>>     oscillation.
>>
>>     If you want to make TCP use for handover situation where one path
>>     might go away or become unusable, it's best to use Multipath TCP
>>     (with coupled congestion control) instead because that works on
>>     the right time scale. Again, I don't think this is a use case for
>>     SR and I would recommend to remove the section entirely.
>>
>>     Mirja
>>
>>
>>     On 05.07.2018 04:08, Gaurav Dawra wrote:
>>>     Hey Alvaro, Mirja,
>>>
>>>     Friendly reminder to further progress this document.
>>>
>>>     Cheers,
>>>
>>>     Gaurav
>>>
>>>     On Mon, Jun 18, 2018 at 5:13 PM, Gaurav Dawra
>>>     <[e-mail] <mailto:[e-mail]>> wrote:
>>>
>>>         Hi Alvaro, Mirja
>>>
>>>         Any feedback or next steps to close this?
>>>
>>>         Cheers,
>>>
>>>         Gaurav
>>>
>>>         Sent from my iPhone
>>>
>>>         On Jun 12, 2018, at 7:06 AM, Gaurav Dawra
>>>         <[e-mail] <mailto:[e-mail]>> wrote:
>>>
>>>>         Hi Mirja,
>>>>
>>>>         Friendly Reminder...Could you please also advice if there
>>>>         is anything further to DISCUSS on this document which was
>>>>         also related to TCP updates below?
>>>>
>>>>         Cheers,
>>>>
>>>>         Gaurav
>>>>
>>>>         On Thu, Jun 7, 2018 at 9:02 AM, Alvaro Retana
>>>>         <[e-mail] <mailto:[e-mail]>> wrote:
>>>>
>>>>             Thanks Martin!
>>>>
>>>>
>>>>             On June 6, 2018 at 3:14:45 PM, Martin Stiemerling
>>>>             ([e-mail] <mailto:[e-mail]>) wrote:
>>>>
>>>>>             Hi Alvaro, all,
>>>>>
>>>>>             Thanks for addressing my concerns.
>>>>>
>>>>>             This version is good to go from my side.
>>>>>
>>>>>             Kind regards,
>>>>>
>>>>>             ;Martin
>>>>>
>>>>>             Am 30.05.18 um 21:55 schrieb Alvaro Retana:
>>>>>             > Martin:
>>>>>             > br/>> Hi!!  How are you?
>>>>>             > br/>> Gaurav just posted a revision. Please takke a
>>>>>             look and let us know if br/>> the changes address your
>>>>>             concerrns or not.
>>>>>             > br/>>
>>>>>             https://www.ietf.org/rfcdiff??url2=draft-ietf-spring-segment-routing-msdc-09
>>>>>             <https://www.ietf.org/rfcdiff??url2=draft-ietf-spring-segment-routing-msdc-09>
>>>>>
>>>>>             > br/>> Thanks!!!
>>>>>             > br/>> Alvaro. <
>>>>>             > br/>> On May 25, 2018 at 12:08:46 PM, Gaurav Dawra
>>>>>             (([e-mail] <mailto:[e-mail]>
>>>>>             br/>> <mailto:gdawra.ietf@
>>>>>             <mailto:gdawra.ietf@>@gmail.com <http://gmail.com>>)
>>>>>             wrote:
>>>>>             > br/>>> Hi Martin, <
>>>>>             >>
>>>>>             >> Thanks for review. I will post the new version.
>>>>>             Hopefully, it will br/>>> address all your comments
>>>>>             and we can close thhis review.
>>>>>             >>
>>>>>             >> Any updates on below response?
>>>>>             >>
>>>>>             >> Cheers,
>>>>>             >>
>>>>>             >> Gaurav
>>>>>             >>
>>>>>             >> Sent from my iPhone
>>>>>             >>
>>>>>             >> On May 23, 2018, at 4:17 AM, Gaurav Dawra
>>>>>             <[e-mail] <mailto:[e-mail]>
>>>>>             br/>>> <mailto:gdawra.ietf@
>>>>>             <mailto:gdawra.ietf@>@gmail.com <http://gmail.com>>>
>>>>>             wrote:
>>>>>             >>
>>>>>             >>> Hi Martin,
>>>>>             >>>
>>>>>             >>> Thanks for the review. Any further comments here ? I
>>>>>             will post the br/>>>> new version soon. <
>>>>>             >>>
>>>>>             >>> Cheers,
>>>>>             >>>
>>>>>             >>> Gaurav
>>>>>             >>>
>>>>>             >>> Sent from my iPhone
>>>>>             >>>
>>>>>             >>> On May 16, 2018, at 7:44 PM, Gaurav Dawra
>>>>>             <[e-mail] <mailto:[e-mail]>
>>>>>             br/>>>> <mailto:gdawra.ietf@
>>>>>             <mailto:gdawra.ietf@>@gmail..com <http://gmail.com>>>
>>>>>             wrote:
>>>>>             >>>
>>>>>             >>>> Hi Martin, 
>>>>>             >>>> 
>>>>>             >>>> Apologies from my end we had few internal authors
>>>>>             conversations on br/>>>>> the points you have raised. <
>>>>>             >>>> 
>>>>>             >>>> Please find below my response. I will be happy to discuss further, br/>>>>> if needed. <
>>>>>             >>>> 
>>>>>             >>>> <Gaurav> inline... 
>>>>>             >>>>
>>>>>             >>>>> On Apr 9, 2018, at 7:58 AM, Martin Stiemerling <[e-mail] <mailto:[e-mail]>
>>>>>             br/>>>>>> <mailto:[e-mail]
>>>>>             <mailto:[e-mail]>>> wrote:
>>>>>             >>>>> 
>>>>>             >>>>> Hi Gaurav, 
>>>>>             >>>>> 
>>>>>             >>>>> This got lost on my end, sorry for this. The filter
>>>>>             just moved br/>>>>>> these messages out of my sight...
>>>>>             :-/
>>>>>             >>>>>
>>>>>             >>>>> Am 15.02.18 um 05:47 schrieb Gaurav Dawra: 
>>>>>             >>>>>> Hey Martin, 
>>>>>             >>>>>> Sorry for late reply. Please see some comments inline[Gaurav]
>>>>>             >>>>>>> On Jan 9, 2018, at 2:25 PM, Martin Stiemerling
>>>>>             br/>>>>>>>> <mls.ietf@@gmail.com <http://gmail.com>
>>>>>             <mailto:[e-mail]
>>>>>             <mailto:[e-mail]>> br/>>>>>>>>;
>>>>>             <mailto:[e-mail]
>>>>>             <mailto:[e-mail]>>> wrote:
>>>>>             >>>>>>> 
>>>>>             >>>>>>> Hi all, 
>>>>>             >>>>>>> 
>>>>>             >>>>>>> I've reviewed this document as part of the transport
>>>>>             area review br/>>>>>>>> team's ongoing effort to
>>>>>             review key IETF documents. These br/>>>&gtt;>>>>
>>>>>             comments were written primarily for the transport area
>>>>>             directors, br/>>>>>>>> but are copied to the
>>>>>             doocument's authors for their information br/>>>>>>>&>
>>>>>             and to allow them to address any issues raised. When
>>>>>             done at the
>>>>>             >>>>>>> time of IETF Last Call, the authors should consider this review br/>>>>>>>> together with any
>>>>>             other last-call comments they receive. Please
>>>>>             br/>>>&>>>>> always CC tsv-art@… if you reply to or
>>>>>             forward this review.
>>>>>             >>>>>>> 
>>>>>             >>>>>>> Summary: 
>>>>>             >>>>>>> This draft has serious issues in Section 7..1, 7.2 and in one part br/>>>>>>>> of Secction3,
>>>>>             described in the review, and needs to be rethought.
>>>>>             br/>>&>>>>>> The other sections are good AFAIK.
>>>>>             >>>>>>> 
>>>>>             >>>>>>> 
>>>>>             >>>>>>> Technicals: 
>>>>>             >>>>>>> The overall draft looks ok, but the three points below look br/>>>>>>>> strange and need a fix
>>>>>             before publication IMHO:
>>>>>             >>>>>>> 
>>>>>             >>>>>>> Both Sections, 7.1. and 7.2., are describing ideas, but not well br/>>>>>>>> proven
>>>>>             funcationality and not even safe to use functionality.
>>>>>             br/>>>&>>>>> Both are some sort discussing that
>>>>>             different paths in the network br/>>>>>>>> could be
>>>>>             used by the eend host traffic. This sounds pretty much
>>>>>             br/>>>>>>&gtt;> like the Path Aware Networking
>>>>>             Proposed Research Group br/>&gtt;>>>>>>
>>>>>             (https://irtf.org/panrg) and hints to the fact that
>>>>>             there is no br/>>>>>>>> commonly understannd and
>>>>>             accepted engineering solution in this space.
>>>>>             >>>>>>> 
>>>>>             >>>>>>> Section 7.1: 
>>>>>             >>>>>>> [KANDULA04] is a really old reference that hasn't been followed br/>>>>>>>> up iin recent times
>>>>>             and even worse there is no evidence that this
>>>>>             br/>&gtt;>>>>>> is going to work good enough or stable
>>>>>             enough under real Internet br/>>>>>>>> traffic.
>>>>>             Additioonally, it is more than unclear how any modern
>>>>>             TCP br/>>>>&ggt;>>> implementation will react to this
>>>>>             >>>>>> [Gaurav] Will get back on this.
>>>>>             >>>>> 
>>>>>             >>>>> I will reply to the other email dicussing this. 
>>>>>             >>>> <Gaurav> I have replied to other thread. 
>>>>>             >>>>> 
>>>>>             >>>>>>> 
>>>>>             >>>>>>> Section 7.2: 
>>>>>             >>>>>>> This section describes an idea without detailing too
>>>>>             much about br/>>>>>>>> any furtther aspects. Further
>>>>>             it changes the commonly accepted br/>>>;>>>>> notion
>>>>>             of what an end host can do with the network. At best
>>>>>             this br/>>>>>>>> would require a good ddefinition of
>>>>>             what an end host in your br/>>>>>>>&ggt; setting is,
>>>>>             e.g., a highly modified piece of (at least) software
>>>>>             >>>>>>> that usually not found in OS availble on the market
>>>>>             (yet?)
>>>>>             >>>>>>> Further communicating instantaneous path
>>>>>             characteristics to a br/>>>>>>>> central point is
>>>>>             potentially a bad idea, as the data is already
>>>>>             br/>>>;>>>>> outdated when reported by any node.
>>>>>             >>>>>> [Gaurav] I believe Authors are trying to highlight that Host which br/>>>>>>> is defineed in
>>>>>             br/>>>>>>>
>>>>>             (https://tools.ietf.org/html/draftt-ietf-spring-segment-routing-15
>>>>>             <https://tools.ietf.org/html/draftt-ietf-spring-segment-routing-15>)
>>>>>             br/>>>>>>> can innfluence the traffic based on the
>>>>>             Calculations locally or br/>>&gtt;>>>> jointly with
>>>>>             the controller. Implementations can decide how much
>>>>>             br/>>>>>>> Centralized -vs- localized coontrol is
>>>>>             allowed at Host based on br/>>>>>>> perfoormance data
>>>>>             collection.
>>>>>             >>>>> 
>>>>>             >>>>> Performance data collection (monitoring?) isn't crucial when it br/>>>>>> comes to timely (actuaally
>>>>>             real-time) reaction. However, this br/>>>>>> secttion
>>>>>             isn't just about performance data collection as it is
>>>>>             about br/>>>>>>> "Performance-aware routing" this
>>>>>             seems to try to interact in br/>>>>>> real-time with
>>>>>             the network behhavior of TCP. This isn't even close
>>>>>             br/>>>>>> to acceeptable.
>>>>>             >>>>> 
>>>>>             >>>>> I would be ok to say that it is useful to collect performance data br/>>>>>> for offline analysis and
>>>>>             improvement of the data network. However,
>>>>>             br/>>>>>&ggt; this is at completely different
>>>>>             magnitues of time scales.
>>>>>             >>>>>
>>>>>             >>>>> I would recommend to remove the TCP part from this section entirely.
>>>>>             >>>> <Gaurav>Ack, will update in next rev: 
>>>>>             >>>> 
>>>>>             >>>> Section will read like this: 
>>>>>             >>>> 
>>>>>             >>>> ; 
>>>>>             >>>> /Knowing the path associated with flows/packets, the end host may/
>>>>>             >>>> /deduce certain characteristics of the path on its own, and/
>>>>>             >>>> /additionally use the information supplied with path information/
>>>>>             >>>> /pushed from the controller or received via pull request. The host/
>>>>>             >>>> /may further share its path observations with the centralized agent,/
>>>>>             >>>> /so that the latter may keep up-to-date network health map to assist/
>>>>>             >>>> /other hosts with this information./ 
>>>>>             >>>> // 
>>>>>             >>>> /For example, an application A.1 at HostA may pin a flow destined/
>>>>>             >>>> /to HostZ via Spine node Node5 using label stack {16005, 16011}. The/
>>>>>             >>>> /application A.1 may collect information on packet loss, deduced from/
>>>>>             >>>> /Other offline mechanisms. [There are some pingMesh mechanisms which /
>>>>>             >>>> /Can be used here]/ 
>>>>>             >>>> /Through these mechanisms information to a centralized agent, e.g./
>>>>>             >>>> /after a flow completes, or periodically for longer lived flows./
>>>>>             >>>> /Next, using both local and/or global performance data, application/
>>>>>             >>>> /A.1 as well as other applications sharing the same resources in the/
>>>>>             >>>> /DC fabric may pick up the best path for the new flow, or update an/
>>>>>             >>>> /existing path (e.g.: when informed of congestion on an existing/
>>>>>             >>>> /path)./ 
>>>>>             >>>> ; 
>>>>>             >>>> 
>>>>>             >>>>> 
>>>>>             >>>>> 
>>>>>             >>>>>>> 
>>>>>             >>>>>>> Section 3, 3rd bullet point: 
>>>>>             >>>>>>> It is the foundation of TCP that the network is
>>>>>             regarded as a br/>>>>>>>> black box and that you infer
>>>>>             from the transmission of packets br/>>>>;>>>> what the
>>>>>             current state of the network path is. Inferring
>>>>>             network br/>>>>>>>> path metrics (you mention SRTT,
>>>>>             MSS, CWND ) is a bad idea, as br/>>>>>>>>; this would
>>>>>             required that all paths exhibit this and if not what
>>>>>             br//>>>>>>>> is going to happen?
>>>>>             >>>>>>> It could be an interesting research field to change many points br/>>>>>>>> in TCP'ss behavior,
>>>>>             but this once again points to the fact that
>>>>>             br/>>>&>>>>> this not the IETF works but IRTF or
>>>>>             elsewhere.
>>>>>             >>>>>> [Gaurav] Martin, Authors are trying to suggest that TCP is rightly br/>>>>>>> treating Network as
>>>>>             Black Box. Authors are implying per path br/>>>>;>>>
>>>>>             performance metrics as not cached. Is there some
>>>>>             change in text br/>>>>>>> you are suggesting??
>>>>>             >>>>> 
>>>>>             >>>>> 
>>>>>             >>>>> I would recommend to remove the 3rd bullet point completey. TCP br/>>>>>> isn't the place to rememmber
>>>>>             "good" or "bad" paths. This is br/>>>>>> something the
>>>>>             network could decide, e.g., rerouting TCP flows
>>>>>             br/>&ggt;>>>> within the network or changing the
>>>>>             forwarding path in the network br/>>>>>> for
>>>>>             particular flows (if it is not routed).
>>>>>             >>>> 
>>>>>             >>>> <Gaurav> Ack, after discussion, we will remove the Section 3 - 3rd br/>>>>> bullet point. Willl
>>>>>             update in next rev - coming shortly.
>>>>>             >>>> 
>>>>>             >>>>> 
>>>>>             >>>>> Kind regards, 
>>>>>             >>>>> 
>>>>>             >>>>>  Martin 
>>>>>             >>>>> 
>>>>>             >>>> 
>>>>
>>>>
>>>
>>>
>>>
>>>     _______________________________________________
>>>     Tsv-art mailing list
>>>     [e-mail] <mailto:[e-mail]>
>>>     https://www.ietf.org/mailman/listinfo/tsv-art
>>>     <https://www.ietf.org/mailman/listinfo/tsv-art>
>>
>