[E-impact] Why carbon aware routing would break the Internet and emit more carbon Re: Carbon aware routing

[Rudolf van der Berg <[e-mail]>]

The very interesting paper on carbon aware routing by Sawsan yesterday, of
which I only caught the last part of the presentation (sorry) inspired me
to write a more general analysis of carbon aware routing, with some
reference to the paper, but also a more fundamental critique of the idea of
carbon aware routing.

*Why carbon aware routing would break the Internet and emit more carbon *

*Routing isn't computer science, it is economics*
The science behind routing protocols is not computer science, it is
economics. There are thousands of actors with varying levels of dependency
upon each other. This is true within one network, but infinitely more so
between thousands of networks. These actors are all boundedly rational and
have varying levels of information on the state of their own and other's
networks in the past and at this moment. To make things more complicated,
their actions influence each other, can be mutually incompatible or
mutually beneficial. They all have different assumptions, what caused their
own actions and how this influenced the actions of others in the past and
in the future. To completely mess things up, even if all data points and
assumption were fully true until that point, that would not allow them to
predict the future state of the network and routing of traffic (not even a
second in the future!). It would be impossible to know whether
their decision is "the most optimal", because routing is part of a
chaotic system, called "the World". The state of the network and the
traffic flowing through it, is not a closed system. It is influenced by
everything that happens outside the network and that can be a lot, varying
from war, natural disasters and pestilence to human error, Taylor Swift and
people helping each other in times of need. That doesn't mean that the
system is completely random, but it can't be relied on to be stable and it
changes have to be expected and dealt with at any time. So the best
routing system is the one that requires the least complete knowledge of the
state of the network and can handle changes to that state at great speed
and with great resilience.

The routing of traffic within one network is  determined by whoever is in
control of that network. No two networks need to be the same, because the
preferences and needs of those who control it may be different. Beautiful
words can be spoken, about aligning the operation of the network and
routing of traffic with lofty goals such as carbon awareness, customer
needs etc. The reality is that there are many constraints and influences
that will make the alignment of the daily operation of the network with
those beautiful words and lofty goals a matter of opinion and not fact.
There is therefore no most optimal way of designing a network or routing
traffic over it. Each network is different and therefore operates
differently. Interconnection of network makes it even more complex to
operate a network optimally.

When it comes to cross domain routing concepts such as path dependent
(SCION),  carbon based, cost based, QoS based routing were all tried in the
1980s and 1990s, failed and  surpassed by BGP4. All those earlier and later
failed routing protocols made the same flawed assumption as carbon free
routing does today; if I know the state of the network, I can choose the
most optimal route for me and the whole world. In the late eighties most
routing protocols required that the state of the network was known and
couldn't change too much. This almost broke the Internet. Fortunately
advances allowed BGP to be developed. It requires remarkably little
knowledge of the state of other networks. It can't see the difference
between an Autonomous System Number of BT vs JANET or Google. It doesn't
know what routers they use, what bandwidth there is, what physical distance
there is etc. It knows how to get from one ASN to another ASN. And despite
not knowing what a router, capacity, a telco, content provider, customer,
supplier, carbon, money or a country border is, it works rather well. It
turns out all those other variables may be very important to individuals
who operate one network, but aren't relevant to inter-domain routing. A
basic explanation I wrote of the economic transactions that take place,can
be found at. https://arstechnica.com/features/2008/09/peering-and-transit/
(This model is different and the opposite of the more often cited, but
flawed, two-sided models of interconnection of Laffont, Marcus, Rey and
Tirole)

*Carbon aware routing in one network needs an accurate model of the
electricity network *
I'm quite sceptical of carbon aware routing within one network and even
more so when it happens across networks. For it to work, the routing
algorithm would need a full understanding of how the exhaust of carbon is
the result of electricity generation. The paper that was presented does not
have such a model as part of the CATE algorithm. The paper says that
national and European electricity markets are interconnected, but the
algorithm doesn't  derive the consequence from it: That the electricity
network in Scotland is to a large extent the same one as in England and is
interconnected with the electricity networks of EU countries. The total
generation is related to the total consumption in the UK and that apart
from losses in transport, import and export and  congestion, the total
electricity consumption of the UK is relevant and not the location.

 Routing traffic through Scotland doesn't make the wind blow faster in
Scotland. It doesn't use less electricity in the UK.  Most low carbon and
zero carbon electricity sources aren't variable with demand. Their yield is
driven by the intensity of the wind, water and the sun. Green electricity
therefore has a priority over fossil fuel based electricity. Fossil fuel is
the back-up for what can't be generated by renewable sources. (there are
some renewable sources that may handle short peaks, eg water for TV-pick up
in the UK) So shutting down an optic from London to Leeds may save
electricity. This good, because it means a bit less electricity is needed
in total and so a little bit less non-renewable fuel needs to be burnt.
However, it will not lead to less carbon emitted, if shutting down a link
to Leeds means that for traffic to Middlesborough a link from Manchester to
Edinburgh and then to Middlesborough needs to be activated. The extra hop
and longer distance for Edinburgh would ctually increase the total
electricity consumption of the UK. This would require more fossil fuel to
be burned. Scotland's green energy is a fixed quantity. Scotland doesn't
become less green when traffic is routed through England. The UK does
become less green if more electricity is needed in total to route traffic
through Scotland. So a carbon aware routing protocol can only work if the
model of the electricity network is complete and accurate for both how
electricity is generated, consumed and distributed.

*What if Scotland was independent from  England, Wales and Northern
Ireland, would that be better?*
Theoretically it could be possible that the electricity grid of Scotland is
completely independent of England, Wales and Northern Ireland. In that case
the electricity consumption and the carbon footprint would be independent
metrics. Scotland would need to have an over supplied and 100% green grid,
completely independent of England, with no interdependence and no
interconnection. If at the same time England has a network that still does
emit CO2 in significant quantities to generate electricity, then there
might be a form of arbitrage possible. In this case it might be possible
that linking parts of BT's network in England, Wales and Northern Ireland
through Scotland results in less carbon emissions by the combined
electricity networks of Scotland, England, Wales and Northern Ireland.

*Carbon aware routing in one network requires an accurate power profile of
each device in the network*
In the paper the authors assume that routers have a linear energy
consumption profile above a base load. This is an academic
oversimplification that is unusable in practice. Routers combine and split
incoming data flows so that they become outgoing data flows. The complexity
of a router is that incoming data flows and outgoing data flows are
completely different from each other in size, distribution and compilation,
though in aggregate they should be exactly the same. As a massive
intersection of roads that should have no traffic jams or crashes. As a
router reaches the maximum capacity of its ports and backplane its energy
consumption is likely to increase in a non-linear fashion, heating it up
more and requiring more cooling. At the same time it increasingly runs
against its limits. Though the likes of Google have shown that they can get
close to the technical limit of their networking equipment in the
datacenter network for their services, it is very clear that this is a
non-trivial task, that required a lot of upfront effort into knowing the
behaviour of all their equipment and of the applications that run over it.
https://cloud.google.com/blog/topics/systems/the-evolution-of-googles-jupiter-data-center-network

In the case of a telecom operator such as BT, the electricity consumption
of the network is less deterministic. BT has a large number of very
different types of users and usages. It has different agreements with those
users. The equipment is at best of 2 different generations; the previous
one and the one it is moving to. Its energy consumption is in part
determined by physical distances, but also by weather! Certainly now
climate change makes it warmer and sunnier in the UK too! The network
changes every hour of every day, due to wear, tear, upgrades, downgrades,
cable breaks, equipment failure etc. All of this isn't known to BT at any
given moment and can take a while to percolate through to configuration
databases at which point the state of the network is already different.

What the CATE algorithm tries to do therefore falls far short of what would
be needed, because it makes linear assumptions and assumes a stable state
of BTs network, while not correctly incorporating the operation of the
electricity network.

*Carbon aware routing across independent networks will break the Internet*
Routing based on BGP makes use of Autonomous Systems and how many hops
there are between the source ASN and the target ASN. Less hops is better.
BGP doesn't make assumptions about what an ASN represents. BGP doesn't know
anything about how many routers, how much capacity, how many miles or how
much traffic is in each ASN. Between the more than 75K ASNs there are
continuous changes and BGP will announce them and algorithms will process
them. It is a dumb, but scalable system, with a remarkably efficient and
scalable network of interconnected networks as an emergent property.

Carbon awareness would require that the routing algorithm actually knows
and understands the electricity network that each piece of equipment in the
entire Internet. Everything that should have been done in the BT network
for the CATE algorithm to have reliable results, but then for every
autonomous system and all the networks comprising those ASNs.

What would really break the Internet however is that if carbon is used as a
metric to guide routing decisions, than certain paths will always emerge as
the most optimal from a carbon perspective. This is clear from the lament
of the authors about the Geant network relying so much on Germany for its
connectivity and Germany having such high carbon emissions. If carbon aware
routing was a thing on the Internet, all traffic would want to evade
networks that run through Germany (or not if they actually understand that
Europe has an interconnected electricity market). What would happen is that
traffic would seek the route with the lowest carbon emissions and all
traffic would be sent to that route, without consideration for capacity,
costs etc. That route would be overloaded, of which it would have to alert
other networks, but those wouldn't be able to determine which network can
and which network shouldn't send its traffic over the saturated, but lowest
carbon, link.

A similar issue was relayed to me by ISOC staff after the great earthquake
in Japan, where most links had broken and therefore limited capacity was
available. Japanese telecom firms had sold more "guaranteed" capacity than
there was capacity available in the affected area. The result was that the
routers wouldn't know what network had priority and therefore transmitted
no traffic at all, despite total traffic being less than what little
capacity was left. After the earthquake businesses were closed and most
people were either in shelters hiding or doing practical things for the
recovery and therefore the network was less in use.

Carbon, QoS and other such metrics are of little use in routing across
autonomous systems. Each network is different and has its own operations.
The success of the Internet is that BGP doesn't have to factor it all in.

*Electricity aware networking not carbon aware routing *
The paper does show a more practical approach limiting the negative impact
of use of resources by networks. By looking at the electricity consumption
of links and interfaces and the amount of traffic, there can be a reason to
turn off certain interfaces and equipment during moments of low use. Even
this is non-trivial, because of non-linear effects, redundancy and the ever
changing state of the network and the performance of equipment. Looking at
operational electricity consumption is probably difficult enough for
network staff to deal with. Embedded carbon of new versus old equipment
might be too complex already. To illustrate, a network may supply
connectivity to a customer through 2 times 2x100Gbps on a redundant route
between two towns. The peaks exceed 100Gbps on the link, but not by much
and traffic growth can be limited. 2 times 400Gbps will then use less
electricity at any moment. This can be calculated and demonstrated. The
embedded carbon of earlier replacement of interfaces, combined with the
network being able to use 2 ports for another customer are much more
complex, particularly when the 100Gbps is used elsewhere in the network.

*Carbon aware placement of datacenters*
Transport losses, congestion and other examples of inefficiency and
scarcity in real world electric networks might make it preferable to build
sites that consume a lot of electricity close to where the generation is
greenest. That however applies to BT's peers and not to BT itself. BT is a
nationwide network. Its electricity consumption primarily follows where
people live and work. That typically is England and not Scotland. So
placing a government or hyperscale datacenter in Glasgow might be preferred
over Slough from a carbon aware electricity generation point of view.
Transmitting photons over optical fibre uses less electricity than the
losses of the electricity network over the same distance.

*Conclusion*
The impact of networking on the planet is significant. Every attempt should
be made to decrease the impact whether it is from the operation of the
network or building, upgrading and replacing it. Using less electricity in
networking is good, because it means that less electricity needs to be
generated, whether it is through sustainable or unsustainable generation.
Making routing decisions depend on assumed carbon emissions is however not
going to lead to less energy consumption, less carbon emessions or better
routing and networking. It will likely break the Internet and emit more
carbon. Better awareness of electricity consumption and the lifetime of
equipment is what the internet community can work on.



Op za 17 feb 2024 16:57 schreef Hesham ElBakoury <[e-mail]>:

> Actually this Seyedali et al paper (from Adrian SCION group) is referenced
> by Salwa et al paper (from Noa group).
> The comment made by Salwa's paper says that SCION paper "*takes advantage
> of the SCION architecture in the context of path-aware networks. It derives
> an estimated*
> *carbon footprint for every inter-domain path and then, end-domains choose
> the paths with the least emissions. This should not be confused with the
> network-wide carbon optimization problem that removes any overlapping
> between end-domains*".
>
> There are many papers about green routing and green traffic engineering.
> As Salwa pointed out in her presentation, each has its own limitations.
>
> Hesham
>
> On Sat, Feb 17, 2024, 6:21 AM Michael Welzl <[e-mail]> wrote:
>
>> … and then there was also this:
>> https://netsec.ethz.ch/publications/papers/green_routing2023.pdf
>>
>> Cheers,
>> Michael
>>
>>
>>
>> On Feb 17, 2024, at 1:37 AM, Eve Schooler <[e-mail]> wrote:
>>
>> Thanks Hesham!
>> The paper first appeared in HotCarbon'22, and was re-published along with
>> many other papers from that workshop in the ACM SIGEnergy Energy
>> Informatics Review this past Oct 2023.
>> --Eve
>>
>> On Fri, Feb 16, 2024 at 3:50 PM Hesham ElBakoury <[e-mail]>
>> wrote:
>>
>>> I also recall that Noa and Eve had a paper on carbon aware networking
>>> https://dl.acm.org/doi/10.1145/3630614.3630618. It is also a good
>>> paper.
>>>
>>> It is worth noting that section 8.6 of Sawdan and Noa paper provides a
>>> comparison with the state of the art. "In summary, this paper looked at
>>> practical current considerations, different to assumptions in previous
>>> works. It estimated similar carbon savings while accounting for more
>>> fine-grained carbon
>>>  intensity data, technical operating considerations of routers, without
>>> assumptions on additional renewable energy deployments."
>>>
>>> Hesham
>>>
>>> On Fri, Feb 16, 2024, 12:47 PM Rudolf van der Berg <
>>> [e-mail]> wrote:
>>>
>>>>
>>>> https://ora.ox.ac.uk/objects/uuid:504d0a87-6742-414f-b2c2-be74f2e5b579/files/sc821gm64z
>>>>
>>>> Op vr 16 feb 2024 20:30 schreef Chris Adams <
>>>> [e-mail]>:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> > We had good presentation today on carbon aware routing. More details
>>>>>> on
>>>>>> > carbon aware routing are provided in this paper:
>>>>>> > https://dl.acm.org/doi/10.1145/3629165
>>>>>>
>>>>>
>>>>> I’m not part of an institution with access to that paper at that url -
>>>>> would a kind soul point me to somewhere I can download it, or share a copy
>>>>> with me? I’m very interested in reading it.
>>>>>
>>>>> Thanks in advance.
>>>>>
>>>>> Chris Adams
>>>>>
>>>>> Executive Director
>>>>>
>>>>> w: thegreenwebfoundation.org
>>>>> e: [e-mail]
>>>>> t: @mrchrisadams
>>>>>
>>>>> German Office
>>>>> Naunynstrasse 40
>>>>> 10999 Berlin
>>>>> Germany
>>>>>
>>>>> See our contact page for more details
>>>>> https://www.thegreenwebfoundation.org/contact/
>>>>>
>>>>> Book a short call with me to discuss something.
>>>>> https://cal.com/mrchrisadams
>>>>>
>>>>>
>>>>> On 16. Feb 2024, at 20:03, Dirk Trossen <dirk.trossen=
>>>>> [e-mail]> wrote:
>>>>>
>>>>> CATS, although compute aware,  could incorporate energy metrics in its
>>>>> approach to traffic steering.
>>>>>
>>>>> In fact,  we had discussed previously in CATS work that I published in
>>>>> IFIP Networking 2021 that utilised a cardinal based WFQ at ingress points
>>>>> to dynamically steer traffic. We're currently looking into the right
>>>>> utility function for energy to replace the compute units we used back in
>>>>> 2021.
>>>>>
>>>>> Dirk
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:*Hesham ElBakoury <[e-mail]>
>>>>> *To:*Toerless Eckert <[e-mail]>
>>>>> *Cc:*E-Impact IETF <[e-mail]>
>>>>> *Date:*2024-02-16 19:03:35
>>>>> *Subject:*Re: [E-impact] Carbon aware routing
>>>>>
>>>>> Tvr is my preference as we discussed upon tvr inception. But I think
>>>>> in today's meeting IRTF was mentioned as the venue for this discussion.
>>>>>
>>>>> Hesham
>>>>>
>>>>> On Fri, Feb 16, 2024, 9:57 AM Toerless Eckert <[e-mail]> wrote:
>>>>>
>>>>>> IRTF not sure, but IETF TVR:
>>>>>>
>>>>>> >From charter:
>>>>>>
>>>>>> "Similarly, network traffic might be routed based on energy costs or
>>>>>> expected user data volumes, which may vary predictably over time in
>>>>>> networks prioritizing green computing and energy efficiency."
>>>>>>
>>>>>> Cheers
>>>>>>     Toerless
>>>>>>
>>>>>> On Fri, Feb 16, 2024 at 09:49:03AM -0800, Hesham ElBakoury wrote:
>>>>>> > We had good presentation today on carbon aware routing. More
>>>>>> details on
>>>>>> > carbon aware routing are provided in this paper:
>>>>>> > https://dl.acm.org/doi/10.1145/3629165
>>>>>> >
>>>>>> > Which IRTF group is suitable to discuss this topic More?
>>>>>> >
>>>>>> > Thanks
>>>>>> > Hesham
>>>>>>
>>>>>> > --
>>>>>> > E-impact mailing list
>>>>>> > [e-mail]
>>>>>> > https://www.ietf.org/mailman/listinfo/e-impact
>>>>>>
>>>>> --
>>>>> E-impact mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/e-impact
>>>>>
>>>>>
>>>>> --
>>>>> E-impact mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/e-impact
>>>>>
>>>> --
>>> E-impact mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/e-impact
>>>
>> --
>> E-impact mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/e-impact
>>
>>
>> --
> E-impact mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/e-impact
>