Re: ECMP and flow label [Re: 6man w.g. last call for <draft-ietf-6man-segment-routing-header-11.txt>]

[Brian E Carpenter <[e-mail]>]

Hi Tom,
On 31/03/2018 13:10, Tom Herbert wrote:
> On Fri, Mar 30, 2018 at 3:50 PM, Brian E Carpenter
> <[e-mail]> wrote:
>> On 31/03/2018 05:38, Joel M. Halpern wrote:
>>> I responded a bit too quickly to Tom's note.
>>> I agree with the later parts of his note.  It is the ECMP behavior of
>>> intermediate routers that cause me concern.
>>> Whether his proposed fix is the right one or not is unclear, as the blog
>>> post I referred to suggested more issues.  (And I apologize for being
>>> unable to find a pointer to the blog post.)
>>
>> OK, but if this is a real issue with our RFCs, and not an implementation
>> defect, we need to fix it,independently of the current draft.
>>
>> More below...
>>
>>> Yours,
>>> Joel
>>>
>>> On 3/30/18 12:29 PM, Joel M. Halpern wrote:
>>>> Sorry for being a bit vague.  The issue with packet reordering when
>>>> using flowlabel for ECMP was discussed in a blog post from a senior
>>>> engineer (at a company other than my own) who as far as I know does not
>>>> participate in the IETF.  He went through a number of issues in that.
>>>>
>>>> There are likely ways to work around the problems.  But since the draft
>>>> does not even refer to the issue, there is a serious gap.
>>>>
>>>> Yours,
>>>> Joel
>>>>
>>>> On 3/30/18 11:59 AM, Tom Herbert wrote:
>>>>> On Fri, Mar 30, 2018 at 7:51 AM, Joel M. Halpern <[e-mail]>
>>>>> wrote:
>>>>>> I do not think this document is ready to be sent to the IETF and IESG
>>>>>> for
>>>>>> final approval.
>>>>>> There are several kinds of problems.
>>>>>>
>>>>>> ECMP1: The document asserts that entropy information is put into the
>>>>>> flow
>>>>>> label.   I wish this worked.  I helped bring this idea forward years
>>>>>> ago,
>>>>>> independent of SRH.  Unfortunately, there are two related problems.
>>>>>> First,
>>>>>> adoption appears to be low.  Second, and more important, there are
>>>>>> reports
>>>>>> from the field that doing thi actually breaks other things and cause
>>>>>> packet
>>>>>> re-ordering under some circumstances.  As a result, vendors are
>>>>>> suggesting
>>>>>> that people turn it off even when it is available.
>>>>>
>>>>> Joel,
>>>>>
>>>>> I assume you're referring to:
>>>>>
>>>>> "If an array of layer-3 adjacencies is bound to the End.X SID, then
>>>>> one entry of the array is selected based on a hash of the packet's
>>>>> header (at least SA, DA, Flow Label)."
>>>>>
>>>>> The problem with flow label isn't that it can cause OOO packets, it's
>>>>> interaction with stateful techniques in intermediate nodes like
>>>>> firewalls and load balancer that require all packets for a flow to
>>>>> consistently hit the same stateful device. When the flow label changes
>>>>> in mid flow, packets might be routed to a different device that
>>>>> doesn't have the flow state and hence are dropped.
>>>>>
>>>>> I imagine that one could argue that SRH is intended to be used in
>>>>> closed domains so the operator could ensure there are no stateful
>>>>> nodes present that would have an issue, but that makes correct
>>>>> operation of the protocol conditional.
>>
>> If it's conditional on "source nodes MUST conform to [RFC6437]",
>> would that be a problem?
>>
> Hi Brian,
> 
> Maybe I'm missing it, but I don't think that RFC6437 plainly says that
> the flow label MUST be persistent for the lifetime of the flow.

The tricky bit is defining "flow" so essentially we fixed on the idea
that a flow is whatever the source host decides is a flow. By default,
it's a given transport session. But imagine, however, that you open
multiple simultaneous TCP sessions to the same server and want them
all to be load-balanced together. (This isn't hard to imagine.)
Then maybe a better heuristic would be to give them all the same flow
label.

So yes, there is wiggle room in the RFC. But if you do what's suggested
as a default, a stateless hash of the 5-tuple, you get a given flow label
for each entire TCP connection.

> There's actually some interesting things that can be done by changing
> the flow label, for instance we did this in Linux for a failing
> connection to reroute the flow through the network. It worked quite
> well, but inevitably someone will hit a device that expects the flow
> label to be consistent. We ended up turning the turn feature off. The
> default behavior in Linux now is that the flow label is consistent for
> the life of TCP connections.
> 
>>>>> IMO the fix for the flow label/ECMP problem needs to include an update
>>>>> to RFC6437 specifying 1) the flow label can only be set by the source,
>>>>> nodes in the path MUST NOT set it,
>>
>> We debated this at great length years ago, and it was explicitly changed
>> to the rules in RFC6437, because the previous rule was unrealistic. But
>> it defines clear rules about what forwarding nodes may do.
>>
> Well, sort of clear at best.
> 
> A forwarding node may set a zero flow label to something non-zero so
> that is a changing flow label. Also, a flow may be routed through
> different devices that could set the flow label, however there's no
> reason to believe they would set the label to the same value.

Ah, right, that could happen. Sounds like a Worst Current Practice.
 
> Section 6.1 has always puzzled to me. I'm not sure why the flow label
> is particularly cited as a dangerous covert channel, I would think
> something like the potential slack space in UDP would have been a much
> greater concern. In any case, the conditions that the flow label could
> be changed for security are not normatively described so I imagine
> someone might assume some leeway.

Yes. We were certainly under the impression that some firewalls at the
time simply cleared the flow label to be on the safe side.

> 
>>>>> 2) if the flow label is set, it
>>>>> MUST be an entropy identifier of the encapsulated transport flow so
>>>>> that (SA, DA, flow label) is a good tuple representation of the flow,
>>
>> Wait, "encapsulated"? The flow label refers to the header it occurs in,
>> not to an encapsulated header, which it knows nothing about.
>>
> "Encapsulated" being used in it's broadest meaning. If there is a
> transport flow in the packet, the flow label respresents that and the
> 3-tuple is useful for ECMP and such.

OK.

>> Ignoring this, that is the intention of RFC6437, apart from not misusing the
>> word 'entropy'. Please point to any text in RFC6437 that isn't clear enough.
>>
> I think the description is clear, however I'm not sure if the
> algorithm in Appendix A is so relevant any more.

It's not. With all due respect to Von Neumann,it's not even a very
good algorithm. But it's non-normative. 

> There are now many
> examples of hashes created from packet headers from which the flow
> label can be derived (we use Jenkin's hash in Linux, most  NICs
> implement Toeplitz, etc.). An even simpler method in TCP is to just
> create a random number for each connection that is stored in the
> connection context from which the flow label is derived.

Sure, if you don't mind adding 20 bits of state.

> 
>>>>> 3) the flow label MUST be the same for all packets sent on a flow
>>>>> including fragments.
>>
>> Yes, there is a problem for fragmented packets if the flow label is set by
>> a forwarding node, as mentioned at https://tools.ietf.org/html/rfc6437#page-7
>> That isn't a problem if the source node sets the flow label, which is of
>> course the primary recommendation. (Therefore, it isn't a problem
>> in the RFC 6438 scenario, ECMP in tunnels.)
>>
>> Should we be adding a BCP to that, requiring this choice:
>>
>>       *  A forwarding node might use the 2-tuple to define a flow in all
>>          cases.  In this case, subsequent load distribution would be
>>          based only on IP addresses.
> 
> Then that doesn't give us flow specific ECMP which is useful for
> tunnels. Using ECMP for flows is well deoployed. The opportunity with
> the flow labels was to give the same functionality without routers
> needing to resort to DPI to find transport headers for the hash.

Sure, absent fragmentation. But with fragmentation, we will always have
problems (and DPI will fail).

If the flow label is zero and a fragment header is present, what else
can you do but use the 2-tuple?

    Brian

> 
> Tom
> 
>> ?
>>     Brian
>>
>>>>> Tom
>>>>>
>>>>>> ECMP2: If, with suitable analysis, we decide that this is actually
>>>>>> safe to
>>>>>> do, then the document needs to include both that analysis and clearly
>>>>>> spelled out requirements that operators MUST verify that all their
>>>>>> routers
>>>>>> support this behavior and that they have enabled this behavior before
>>>>>> turning on SRv6 usage.
>>>>>>
>>>>>> Structure: The text in section 2.3 says "It is assumed in this
>>>>>> document that
>>>>>> the SRH is added to the packet by its source".  This is at best
>>>>>> disingenuous.  It is very clear that the value of this behavior lies
>>>>>> primarily in use by the network, not by the packet sources.  Claiming
>>>>>> otherwise results in a document with minimal utility.  Even this
>>>>>> document
>>>>>> itself disagrees with this assertion.  Tucked into the very next
>>>>>> section is
>>>>>> text saying that "outer header with an SRH applied to the incoming
>>>>>> packet".
>>>>>> If this behavior were a clearly spelled out requirement, rather than a
>>>>>> "typically" and if the text in 2.3 were replaced with something
>>>>>> realistic,
>>>>>> then the document would at least be itnernally consistent and match the
>>>>>> expected usages.
>>>>>>
>>>>>> Edge filtering: The text on edge filtering does not actually state that
>>>>>> prevention of packets with SRH and a current DA of an internal node is
>>>>>> mandatory.  Unless it is clearly stated, the security considerations
>>>>>> text as
>>>>>> currently written is significantly weakened.  If it is mandatory,
>>>>>> then again
>>>>>> the deployment section needs to note that an operator needs to verify
>>>>>> that
>>>>>> all of his edge devices support such filtering and have it properly
>>>>>> enabled
>>>>>> in order to use SRv6.
>>>>>>
>>>>>> Edge Filtering and hybrids: Other documents have talked about allowing
>>>>>> external packets with SRv6 entries pointing to internal nodes (which
>>>>>> means
>>>>>> the DA upon arrival at the operator edge will be an internal node as I
>>>>>> understand it).  It the intention is to permit that with appropriate
>>>>>> security, then the edge filtering requirements need to be clear about
>>>>>> the
>>>>>> requirements for cryptographic validation at the edge.
>>>>>>
>>>>>> Yours,
>>>>>> Joel
>>>>>>
>>>>>>
>>>>>> On 3/29/18 4:30 PM, Bob Hinden wrote:
>>>>>>>
>>>>>>>
>>>>>>> This message starts a two week 6MAN Working Group Last Call on
>>>>>>> advancing:
>>>>>>>
>>>>>>>          Title           : IPv6 Segment Routing Header (SRH)
>>>>>>>          Authors         : Stefano Previdi
>>>>>>>                            Clarence Filsfils
>>>>>>>                            John Leddy
>>>>>>>                            Satoru Matsushima
>>>>>>>                            Daniel Voyer
>>>>>>>          Filename       : draft-ietf-6man-segment-routing-header-11.txt
>>>>>>>          Pages          : 34
>>>>>>>          Date           : 2018-03-28
>>>>>>>
>>>>>>>
>>>>>>> https://tools.ietf.org/html/draft-ietf-6man-segment-routing-header
>>>>>>>
>>>>>>> as a Proposed Standard.  Substantive comments and statements of support
>>>>>>> for publishing this document should be directed to the mailing list.
>>>>>>> Editorial suggestions can be sent to the author.  This last call will
>>>>>>> end on 12 April 2018.
>>>>>>>
>>>>>>> An issue tracker will be setup to track issues raised on this document.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Bob & Ole
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --------------------------------------------------------------------
>>>>>>> IETF IPv6 working group mailing list
>>>>>>> [e-mail]
>>>>>>> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>>>>>>> --------------------------------------------------------------------
>>>>>>>
>>>>>>
>>>>>> --------------------------------------------------------------------
>>>>>> IETF IPv6 working group mailing list
>>>>>> [e-mail]
>>>>>> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>>>>>> --------------------------------------------------------------------
>>>>
>>>> --------------------------------------------------------------------
>>>> IETF IPv6 working group mailing list
>>>> [e-mail]
>>>> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>>>> --------------------------------------------------------------------
>>>>
>>>
>>> --------------------------------------------------------------------
>>> IETF IPv6 working group mailing list
>>> [e-mail]
>>> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>>> --------------------------------------------------------------------
>>>
>>
>> --------------------------------------------------------------------
>> IETF IPv6 working group mailing list
>> [e-mail]
>> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>> --------------------------------------------------------------------
>