Re: [IPv6] Removing extension headers from packets (revisited)

[Mark Smith <[e-mail]>]

Hi Tom,

On Thu, 17 Aug 2023 at 00:54, Tom Herbert <[e-mail]> wrote:
>

<snip>

>>
>> Tom,
>>
>> Why is it not our job to design tools that operators can’t shoot themselves in the foot with?
>
>
> Hi Mike,
>
> It's already common that operators take liberties with protocols and already do many things that are non-conformant.

Can you please provide some examples?

Having been a network operator for a long time, and for a number of
networks, in my experience and to my knowledge, it is not common for
operators to "take liberties with protocols and already do many things
that are non-conformant".

The main reason is that they can cause unreliable and unavailable
services, which cause customers to complain, and then eventually
cancel their services.

> We're already pretty good designing tools that they can shoot themselves in the foot with!
>>
>>
>> If it’s trivial to misconfigure routers that then allow “limited domain only” packets to leak out onto the internets then that’s not a failure of the operator but a failure of the design and implementation of the protocol.
>
>
> That's a problem that already exists. For instance, it's a bad idea for segment routing packets to escape the segment routing domain-- this is such a major concern that there is a proposal to use a new EtherType just for segment routing to make filtering easier -- that would be replete with opportunities for operators to shoot themselves in the foot :-)
>

I don't think you've understood the motivation for that proposal.

It will be inspired by MPLS not leaking onto the Internet, because it
has a different Ethertype than IPv4 or IPv6, and MPLS isn't required
to be enabled within a network, between networks and to end-user
devices to carry end-user Internet traffic.

A lot of the risks that SRv6 is creating are because end-users use
IPv6 for their Internet traffic too, meaning end-user devices can be
used to spoof SRv6 IPv6 packets. This risk just doesn't exist with
networks using different Ethertype MPLS (or SRoMPLS).


>
>>
>>
>> In my other field of expertise we did a study on an intensive care ward looking at drug errors and discovered that 1) there was a surprising number of them (thousands of errors per month) and 2) that a huge amount of them were caused by the way we were storing the drugs on the dispensing trolleys - different concentrations of epinephrine placed next to each other etc.
>> We then “engineered” the ability to make those errors out of the system and as a direct result the “operators” weren’t able to make those mistakes leading to so much less harm.
>> (I believe death by doctor is running at about 400k people per year in the US so still loads of work to do)
>
>
> Yes, it's a problem. But we also have to consider the benefits versus potential harm. If a packet with Hop-by-Hop options escapes a limited domain into the Internet, then the most likely outcome based on current data is that the packet will be dropped. If Hop-by-Hop options are removed then the packet has better chances to successfully traverse the Internet. If there is some misconfiguration, then the likely effect is that packet is dropped anyway. So in this case, I don't see that the potential harm is outweighs the potential benefit.
>

For a network operator, packets being dropped isn't the end of the story.

Packets are sent for a reason, so when they're dropped, somebody
usually and eventually will want to know why, because most of the time
it is causing a performance problem or a service outage.

If your protocol specification or implementation makes finding out why
harder than necessary, then it's less likely to be adopted and
deployed by network operators, who are dealing directly with the
end-users of the IETF protocols.

"... the IAB believes that, when there is a conflict between the
interests of end users of the Internet and other parties, IETF
decisions should favor end users." - RFC 8890, "The Internet is for
End Users".

The longer a fault takes to rectify, the angrier end-users (customers) get.

>>
>> Blaming operators misconfiguration for ie all the RFC1918 being blocked at my borders is victim shaming when the actual blame lies with the engineers that designed the protocols and the vendors that implement them.
>>
>> So as we haven’t puled the trigger on all these new wonderful ways for “limited domain only” packets to escape out in the wild with completely unknown consequences when it hits all the stuff still running 20 year old network code, why don’t we try and find solutions that don’t rely on operators being infallible?
>
>
> As I mentioned, the problem already exists and solutions are in deployment in terms of trying to contain packets or data in limited domains.

Be conservative with what you send, ...

"Prevention is better than better than cure." - Erasmus.

Regards,
Mark.


>
> Tom
>
>
>>
>> Coz y’all are clever but none of us are gods.
>>
>> Mike
>>
>> >> ?
>> >>
>> >> These and other problems caused by EH removal were covered in the
>> >> "In-Flight EH Insertion Considered Harmful draft in the section about
>> >> failed EH removal.
>> >
>> > EH insertion would be considered harmful since it increases packet
>> > size and could exceed path MTU to the point such that a sender
>> > couldn't even send minimal size MTU packets without them being
>> > systematically dropped. So EH insertion can materially break the IPv6
>> > protocol, but I still don't see how EH removal could break it.
>> >
>> > Tom
>> >
>> >>
>> >> https://datatracker.ietf.org/doc/html/draft-smith-6man-in-flight-eh-insertion-harmful-02
>> >>
>> >> I presented on it at IETF 106, slide deck:
>> >>
>> >> https://datatracker.ietf.org/meeting/106/materials/slides-106-6man-sessb-in-flight-ipv6-extension-header-insertion-considered-harmful-00
>> >>
>> >> It was suggested that it be worked towards publishing as a 6man WG
>> >> RFC, however quite frankly, I could tell it was going to be ignored by
>> >> the SPRING working group, and it was, so I didn't bother spending any
>> >> further time on it.
>> >>
>> >>
>> >> Regards,
>> >> Mark.
>> >>
>> >>
>> >>
>> >>> Tom
>> >>>
>> >>>
>> >>>> Yours,
>> >>>>
>> >>>> Joel
>> >>>>
>> >>>> On 8/14/2023 10:52 PM, Tom Herbert wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Mon, Aug 14, 2023, 6:12 PM Joel Halpern <[e-mail]> wrote:
>> >>>>>
>> >>>>> Yes, knowing what the correct egress should be is, for the general case, a harder problem.  But, the example you cited is segment routing.  If the host is trusted enough by the operator to actually provide a segment route across the operator domain, it seems necessary for the host to have enough information to properly address the egress.  Thus, for the case you chose for an example, encaps seems quite natural, efficient, and achievable.
>> >>>>
>> >>>>
>> >>>> Okay, maybe segment routing wasn't the best example :-) Consider draft-kaippallimalil-tsvwg-media- hdr-wireless. The basic idea of this draft is to let hosts annotate packets with information, signals that can be consumed by the network to provide QoS. This is for 3GPP networks, so the hosts are potentially all the mobile devices in a network. The signals are only processed by one intermediate device in the path which is the Mobile Router (basically the first hop router). After that point, the signal is no longer processed. All the mobile hosts have is a default route to their carrier and a destination address, they don't have any knowledge of whether a packet will leave a limited domain and what the egress point-- and, in fact, we don't want them to have that information since it would expose information about the internals of the provider network.
>> >>>>>
>> >>>>>
>> >>>>> There may be other cases where knowing the proper egress is harder, but it seems to me that solving the information problem is MUCH easier than solving the problem of removing bytes from the middle of a packet at wire rate at a high speed inter-operator
>> >>>>
>> >>>>
>> >>>> Functionally, it's equivalent to a decap/encap operation, or a push/pop of headers. Maybe fixed function hardware can't easily do this, but I'd be surprised if programmable devices like P4 couldn't do it. Also, like in the case of draft-kaippallimalil-tsvwg-media- hdr-wireless, the last intermediate that processes thh HBH options could also be the one that removes it from the packet.
>> >>>>
>> >>>> In any case, the points about debuggability and feasibility are well taken, but those are operational concerns. I am still wondering if removing a HBH extension header could actually break the IPv6 protocol.
>> >>>>
>> >>>> Tom
>> >>>>
>> >>>>
>> >>>>> interface.
>> >>>>>
>> >>>>> Yours,
>> >>>>>
>> >>>>> Joel
>> >>>>>
>> >>>>> On 8/14/2023 9:05 PM, Tom Herbert wrote:
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Aug 14, 2023, 5:53 PM Joel Halpern <[e-mail]> wrote:
>> >>>>>>
>> >>>>>> Hmmm.  Trimmed to focus on a point you make.
>> >>>>>>
>> >>>>>> Comments below.
>> >>>>>>
>> >>>>>> Joel
>> >>>>>>
>> >>>>>> On 8/14/2023 8:45 PM, Tom Herbert wrote:
>> >>>>>>>
>> >>>>>>> Hi Mark,
>> >>>>>>>
>> >>>>>>> No, it's not about bandwidth. It's about survivability and
>> >>>>>>> deployability of extension headers. The current prevailing belief is
>> >>>>>>> that extension headers are not usable across the Internet since there
>> >>>>>>> is a high probability of packet drop. However, within a limited
>> >>>>>>> domain, extension headers are deployable, especially if the network
>> >>>>>>> itself is consuming them as would be the case of HBH options
>> >>>>>>> requesting QoS or segment routing headers.
>> >>>>>>>
>> >>>>>>> This presents two problems:
>> >>>>>>>
>> >>>>>>> 1) Hosts create extension headers and if extension headers are known
>> >>>>>>> to work in a limited domain then they can safely use them _if_ the
>> >>>>>>> destination is in the same domain as the source and the entire path is
>> >>>>>>> within the same limited domain. The problem is that a host wouldn't
>> >>>>>>> necessarily know which destination are in the limited domain and which
>> >>>>>>> aren't
>> >>>>>>> 2) Even if a host knows it is sending to a destination outside of the
>> >>>>>>> limited domain, it might still want to apply the packets and services
>> >>>>>>> in extension headers for the time that the packet is in the limited
>> >>>>>>> domain. For instance, segment routing might be used to route a packet
>> >>>>>>> to a specific egress node. At the egress node, the SRH is removed and
>> >>>>>>> the packet can be forwarded into the Internet (I believe this was the
>> >>>>>>> scenario that motivated removal of SRH when it was presented a while
>> >>>>>>> back)
>> >>>>>>>
>> >>>>>>> The alternative for #2 that is usually suggested is to use tunneling
>> >>>>>>> from the source to an egress, for instance the outside IP header could
>> >>>>>>> contain the extension headers and the encapsulated packet is the one
>> >>>>>>> sent one the Internet with no extensions. That might work and be
>> >>>>>>> conformant, but it's significantly more complex and probably not as
>> >>>>>>> practical as just removing extension headers at the border router
>> >>>>>>> (more decisions, we have to explicitly route to egress points, more on
>> >>>>>>> the wire overhead, tracing ICMP errors back to a connection is
>> >>>>>>> tricky). Also, troubleshooting still can be difficult because the same
>> >>>>>>> packet looks very different depending on where it's viewed (for
>> >>>>>>> instance, at one point the packet is encapsulated, at another point
>> >>>>>>> it's not).
>> >>>>>>>
>> >>>>>>> Tom
>> >>>>>>> ---------------------------------------------------
>> >>>>>>
>> >>>>>> <jmh>   You state that encaps / decaps is "significantly more complex
>> >>>>>> and probably not as practical as just removing the extension headers at
>> >>>>>> the border rotuer."  I am missing something .  As far as I can tell,
>> >>>>>> decapsulation at an addressed egress is easy to implement and widely
>> >>>>>> supported in existing hardware. Removing a set of bytes from the middle
>> >>>>>> of a packet is not widely implemented, hard to do in fast path hardware,
>> >>>>>> and prone to interesting failures all its own.   I would have exactly
>> >>>>>> reversed the comparison you make. </jmh>
>> >>>>>
>> >>>>>
>> >>>>> Joel,
>> >>>>>
>> >>>>> Easy at the decapsulation, not necessarily at the encap point. It's the same problem I mentioned before, hosts wouldn't know when a packet will be leaving the domain, and even if they did they wouldn't easily know what the endpoint address of the tunnel is should be for the real destination.
>> >>>>>
>> >>>>> Tom
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>>
>> >>> --------------------------------------------------------------------
>> >>> IETF IPv6 working group mailing list
>> >>> [e-mail]
>> >>> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>> >>> --------------------------------------------------------------------
>> >
>> > --------------------------------------------------------------------
>> > IETF IPv6 working group mailing list
>> > [e-mail]
>> > Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>> > --------------------------------------------------------------------