Re: ECMP and flow label [Re: 6man w.g. last call for <draft-ietf-6man-segment-routing-header-11.txt>]

[Tom Herbert <[e-mail]>]

On Fri, Mar 30, 2018 at 3:50 PM, Brian E Carpenter
<[e-mail]> wrote:
> On 31/03/2018 05:38, Joel M. Halpern wrote:
>> I responded a bit too quickly to Tom's note.
>> I agree with the later parts of his note.  It is the ECMP behavior of
>> intermediate routers that cause me concern.
>> Whether his proposed fix is the right one or not is unclear, as the blog
>> post I referred to suggested more issues.  (And I apologize for being
>> unable to find a pointer to the blog post.)
>
> OK, but if this is a real issue with our RFCs, and not an implementation
> defect, we need to fix it,independently of the current draft.
>
> More below...
>
>> Yours,
>> Joel
>>
>> On 3/30/18 12:29 PM, Joel M. Halpern wrote:
>>> Sorry for being a bit vague.  The issue with packet reordering when
>>> using flowlabel for ECMP was discussed in a blog post from a senior
>>> engineer (at a company other than my own) who as far as I know does not
>>> participate in the IETF.  He went through a number of issues in that.
>>>
>>> There are likely ways to work around the problems.  But since the draft
>>> does not even refer to the issue, there is a serious gap.
>>>
>>> Yours,
>>> Joel
>>>
>>> On 3/30/18 11:59 AM, Tom Herbert wrote:
>>>> On Fri, Mar 30, 2018 at 7:51 AM, Joel M. Halpern <[e-mail]>
>>>> wrote:
>>>>> I do not think this document is ready to be sent to the IETF and IESG
>>>>> for
>>>>> final approval.
>>>>> There are several kinds of problems.
>>>>>
>>>>> ECMP1: The document asserts that entropy information is put into the
>>>>> flow
>>>>> label.   I wish this worked.  I helped bring this idea forward years
>>>>> ago,
>>>>> independent of SRH.  Unfortunately, there are two related problems.
>>>>> First,
>>>>> adoption appears to be low.  Second, and more important, there are
>>>>> reports
>>>>> from the field that doing thi actually breaks other things and cause
>>>>> packet
>>>>> re-ordering under some circumstances.  As a result, vendors are
>>>>> suggesting
>>>>> that people turn it off even when it is available.
>>>>
>>>> Joel,
>>>>
>>>> I assume you're referring to:
>>>>
>>>> "If an array of layer-3 adjacencies is bound to the End.X SID, then
>>>> one entry of the array is selected based on a hash of the packet's
>>>> header (at least SA, DA, Flow Label)."
>>>>
>>>> The problem with flow label isn't that it can cause OOO packets, it's
>>>> interaction with stateful techniques in intermediate nodes like
>>>> firewalls and load balancer that require all packets for a flow to
>>>> consistently hit the same stateful device. When the flow label changes
>>>> in mid flow, packets might be routed to a different device that
>>>> doesn't have the flow state and hence are dropped.
>>>>
>>>> I imagine that one could argue that SRH is intended to be used in
>>>> closed domains so the operator could ensure there are no stateful
>>>> nodes present that would have an issue, but that makes correct
>>>> operation of the protocol conditional.
>
> If it's conditional on "source nodes MUST conform to [RFC6437]",
> would that be a problem?
>
Hi Brian,

Maybe I'm missing it, but I don't think that RFC6437 plainly says that
the flow label MUST be persistent for the lifetime of the flow.
There's actually some interesting things that can be done by changing
the flow label, for instance we did this in Linux for a failing
connection to reroute the flow through the network. It worked quite
well, but inevitably someone will hit a device that expects the flow
label to be consistent. We ended up turning the turn feature off. The
default behavior in Linux now is that the flow label is consistent for
the life of TCP connections.

>>>> IMO the fix for the flow label/ECMP problem needs to include an update
>>>> to RFC6437 specifying 1) the flow label can only be set by the source,
>>>> nodes in the path MUST NOT set it,
>
> We debated this at great length years ago, and it was explicitly changed
> to the rules in RFC6437, because the previous rule was unrealistic. But
> it defines clear rules about what forwarding nodes may do.
>
Well, sort of clear at best.

A forwarding node may set a zero flow label to something non-zero so
that is a changing flow label. Also, a flow may be routed through
different devices that could set the flow label, however there's no
reason to believe they would set the label to the same value.

Section 6.1 has always puzzled to me. I'm not sure why the flow label
is particularly cited as a dangerous covert channel, I would think
something like the potential slack space in UDP would have been a much
greater concern. In any case, the conditions that the flow label could
be changed for security are not normatively described so I imagine
someone might assume some leeway.

>>>> 2) if the flow label is set, it
>>>> MUST be an entropy identifier of the encapsulated transport flow so
>>>> that (SA, DA, flow label) is a good tuple representation of the flow,
>
> Wait, "encapsulated"? The flow label refers to the header it occurs in,
> not to an encapsulated header, which it knows nothing about.
>
"Encapsulated" being used in it's broadest meaning. If there is a
transport flow in the packet, the flow label respresents that and the
3-tuple is useful for ECMP and such.

> Ignoring this, that is the intention of RFC6437, apart from not misusing the
> word 'entropy'. Please point to any text in RFC6437 that isn't clear enough.
>
I think the description is clear, however I'm not sure if the
algorithm in Appendix A is so relevant any more. There are now many
examples of hashes created from packet headers from which the flow
label can be derived (we use Jenkin's hash in Linux, most  NICs
implement Toeplitz, etc.). An even simpler method in TCP is to just
create a random number for each connection that is stored in the
connection context from which the flow label is derived.

>>>> 3) the flow label MUST be the same for all packets sent on a flow
>>>> including fragments.
>
> Yes, there is a problem for fragmented packets if the flow label is set by
> a forwarding node, as mentioned at https://tools.ietf.org/html/rfc6437#page-7
> That isn't a problem if the source node sets the flow label, which is of
> course the primary recommendation. (Therefore, it isn't a problem
> in the RFC 6438 scenario, ECMP in tunnels.)
>
> Should we be adding a BCP to that, requiring this choice:
>
>       *  A forwarding node might use the 2-tuple to define a flow in all
>          cases.  In this case, subsequent load distribution would be
>          based only on IP addresses.

Then that doesn't give us flow specific ECMP which is useful for
tunnels. Using ECMP for flows is well deoployed. The opportunity with
the flow labels was to give the same functionality without routers
needing to resort to DPI to find transport headers for the hash.

Tom

> ?
>     Brian
>
>>>> Tom
>>>>
>>>>> ECMP2: If, with suitable analysis, we decide that this is actually
>>>>> safe to
>>>>> do, then the document needs to include both that analysis and clearly
>>>>> spelled out requirements that operators MUST verify that all their
>>>>> routers
>>>>> support this behavior and that they have enabled this behavior before
>>>>> turning on SRv6 usage.
>>>>>
>>>>> Structure: The text in section 2.3 says "It is assumed in this
>>>>> document that
>>>>> the SRH is added to the packet by its source".  This is at best
>>>>> disingenuous.  It is very clear that the value of this behavior lies
>>>>> primarily in use by the network, not by the packet sources.  Claiming
>>>>> otherwise results in a document with minimal utility.  Even this
>>>>> document
>>>>> itself disagrees with this assertion.  Tucked into the very next
>>>>> section is
>>>>> text saying that "outer header with an SRH applied to the incoming
>>>>> packet".
>>>>> If this behavior were a clearly spelled out requirement, rather than a
>>>>> "typically" and if the text in 2.3 were replaced with something
>>>>> realistic,
>>>>> then the document would at least be itnernally consistent and match the
>>>>> expected usages.
>>>>>
>>>>> Edge filtering: The text on edge filtering does not actually state that
>>>>> prevention of packets with SRH and a current DA of an internal node is
>>>>> mandatory.  Unless it is clearly stated, the security considerations
>>>>> text as
>>>>> currently written is significantly weakened.  If it is mandatory,
>>>>> then again
>>>>> the deployment section needs to note that an operator needs to verify
>>>>> that
>>>>> all of his edge devices support such filtering and have it properly
>>>>> enabled
>>>>> in order to use SRv6.
>>>>>
>>>>> Edge Filtering and hybrids: Other documents have talked about allowing
>>>>> external packets with SRv6 entries pointing to internal nodes (which
>>>>> means
>>>>> the DA upon arrival at the operator edge will be an internal node as I
>>>>> understand it).  It the intention is to permit that with appropriate
>>>>> security, then the edge filtering requirements need to be clear about
>>>>> the
>>>>> requirements for cryptographic validation at the edge.
>>>>>
>>>>> Yours,
>>>>> Joel
>>>>>
>>>>>
>>>>> On 3/29/18 4:30 PM, Bob Hinden wrote:
>>>>>>
>>>>>>
>>>>>> This message starts a two week 6MAN Working Group Last Call on
>>>>>> advancing:
>>>>>>
>>>>>>          Title           : IPv6 Segment Routing Header (SRH)
>>>>>>          Authors         : Stefano Previdi
>>>>>>                            Clarence Filsfils
>>>>>>                            John Leddy
>>>>>>                            Satoru Matsushima
>>>>>>                            Daniel Voyer
>>>>>>          Filename       : draft-ietf-6man-segment-routing-header-11.txt
>>>>>>          Pages          : 34
>>>>>>          Date           : 2018-03-28
>>>>>>
>>>>>>
>>>>>> https://tools.ietf.org/html/draft-ietf-6man-segment-routing-header
>>>>>>
>>>>>> as a Proposed Standard.  Substantive comments and statements of support
>>>>>> for publishing this document should be directed to the mailing list.
>>>>>> Editorial suggestions can be sent to the author.  This last call will
>>>>>> end on 12 April 2018.
>>>>>>
>>>>>> An issue tracker will be setup to track issues raised on this document.
>>>>>>
>>>>>> Thanks,
>>>>>> Bob & Ole
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --------------------------------------------------------------------
>>>>>> IETF IPv6 working group mailing list
>>>>>> [e-mail]
>>>>>> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>>>>>> --------------------------------------------------------------------
>>>>>>
>>>>>
>>>>> --------------------------------------------------------------------
>>>>> IETF IPv6 working group mailing list
>>>>> [e-mail]
>>>>> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>>>>> --------------------------------------------------------------------
>>>
>>> --------------------------------------------------------------------
>>> IETF IPv6 working group mailing list
>>> [e-mail]
>>> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>>> --------------------------------------------------------------------
>>>
>>
>> --------------------------------------------------------------------
>> IETF IPv6 working group mailing list
>> [e-mail]
>> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>> --------------------------------------------------------------------
>>
>
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [e-mail]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------