Re: [E-impact] Why carbon aware routing would break the Internet and emit more carbon Re: Carbon aware routing

[Noa Zilberman <[e-mail]>]

Hi Rudolf,

When we wrote the paper our main goal was to inform the discussion around metrics, and to explore the potential benefits of different approaches (as a follow up on our earlier work<https://eng.ox.ac.uk/media/11756/zilberman22hotcarbon.pdf>). We have considered both energy  related metrics and carbon related metrics.
We also hoped to highlight the effect of static (and dynamic) power and the implications of different traffic patterns.

To specifically address your questions,

  1.
As we are no grid experts, we have engaged with different grid experts about carbon intensity and how it can (or cannot) be used. It would be fair to say that there are different views on the topic, from "a sea of electricity" to the regional model that considers the capacity of the high voltage transmission lines across regions (and losses). As our goal was to explore the effect of carbon intensity differences (which can also be due to local generation), we have adopted the regional carbon intensity model of the National Grid ESO.
  2.
Figures 9 and 10 show both energy and carbon savings. We did not test turning off ports with all possible metrics, just with the ones that have shown the greatest effect (without changing ports). The goal was to explore the difference it makes to control a greater proportion of the power (Figure 6). We also clearly state that a similar methodology can be applied to other (intra-domain) protocols.

For static/dynamic power, I'd like to turn the attention to a presentation<https://eng.ox.ac.uk/media/11vdkdtb/itzikk_evolution-of-switches-power-consumption.pdf> <https://eng.ox.ac.uk/media/11vdkdtb/itzikk_evolution-of-switches-power-consumption.pdf> by Broadcom in the Carbon Aware Networks workshop, which explains evolution and mechanisms for power efficiency in switch ASIC, and puts a number to the dynamic power of one of their ASICs. It sort of gives an upper bound on the potential dynamic power of a router. Obviously this number would change between vendors and device families, but we can work with it to explore how static/dynamic ratio can be improved.

To go back to my initial comment about metrics, as people have pointed out in the thread, one important question is what information are ISPs willing to share. My personal hope was that metrics such as energy labels would be useful, because then an ISP can say "all my equipment is rated A*" or "90% of the routers are A* and 10% are A". Unfortunately this metric didn't turn out to be as useful for routing, though it still has other benefits (e.g., energy efficiency). It is clear that ISPs will not share detailed information about their routers, location and local carbon intensity - not only there's proprietary information involved, but also there are aspects of security (and others). We therefore need to find better solutions both for useful metrics to share and how to share them - especially for inter-domain routing.

Kind Regards,
Noa
________________________________
From: Rudolf van der Berg <[e-mail]>
Sent: 21 February 2024 09:09
To: Noa Zilberman <[e-mail]>
Cc: E-Impact IETF <[e-mail]>
Subject: Re: [E-impact] Why carbon aware routing would break the Internet and emit more carbon Re: Carbon aware routing

To clarify
How CATE performs to these other algorithms on electricity metrics instead of carbon

Op wo 21 feb 2024 09:14 schreef Rudolf van der Berg <[e-mail]<mailto:[e-mail]>>:
Dear professor Zilberman,



Op ma 19 feb 2024 16:27 schreef Noa Zilberman <[e-mail]<mailto:[e-mail]>>:
Dear Rudolf,

Sorry you missed Sawsan’s full presentation. I’d encourage you to read the full paper (paper<https://eng.ox.ac.uk/media/jwpbeeab/elzahr23benefits.pdf>, slides<https://datatracker.ietf.org/doc/slides-interim-2024-eimpact-02-sessa-sawsan-carbon-aware-routing/>) as your definition of and comments on carbon-aware routing don’t align with the scope or details of Sawsan’s paper.


I apologised for not being able to attend the presentation and asking  questions then.

I read the paper before and I read it again. There are two main points of criticism I have;

1. The paper doesn't address that Scotland is an integral part of the UK and EU energy markets. The paper doesn't explain why routing traffic to Scotland is better than BT buying Scottish green electricity, having it transported to England over the interconnectors and using that in Leeds. Or alternatively what happens if BT starts using green electricity in Scotland when that electricity has already been sold on the day ahead market. BT says it is 100% green, though I haven't checked whether that is on a 24x7 basis as Google is aiming to do. This is particularly odd, because the laser says it wants to deal with near real time consumption and public data only has 90 minute granularity.
2. Intra domain routing is what the paper limits itself too. The paper mentions that others have done electricity aware QoS in OSPF and lists them. It then doesn't show how CATE performs on an electricity level, but goes straight to carbon. In addition there has been a ton of academic work on "(something, something) optimised OSPF ", because clearly the IETF had forgotten it and it would make everything much better. Why hasn't BT done electricity aware OSPF until now? There were many proposals, such as mentioned in sources 12 to 17.  Indeed, why has there been only limited uptake of "aware" OSPF in operational networks? Might it be that the operational benefits of "aware" networking are less than the academic work suggests?

I read many papers on routing, its economics and energy consumption. I certainly read your paper.

(Could you please make sure the Telefonica reference isn't used in future papers? It uses kWh/GB!)

Kind regards,

Rudolf



Kind Regards,
Noa


________________________________
From: E-impact <[e-mail]<mailto:[e-mail]>> on behalf of Rudolf van der Berg <[e-mail]<mailto:[e-mail]>>
Sent: 17 February 2024 21:53
To: Hesham ElBakoury <[e-mail]<mailto:[e-mail]>>
Cc: Michael Welzl <[e-mail]<mailto:[e-mail]>>; eve.schooler <[e-mail]<mailto:[e-mail]>>; E-Impact IETF <[e-mail]<mailto:[e-mail]>>
Subject: [E-impact] Why carbon aware routing would break the Internet and emit more carbon Re: Carbon aware routing

The very interesting paper on carbon aware routing by Sawsan yesterday, of which I only caught the last part of the presentation (sorry) inspired me to write a more general analysis of carbon aware routing, with some reference to the paper, but also a more fundamental critique of the idea of carbon aware routing.

Why carbon aware routing would break the Internet and emit more carbon

Routing isn't computer science, it is economics
The science behind routing protocols is not computer science, it is economics. There are thousands of actors with varying levels of dependency upon each other. This is true within one network, but infinitely more so between thousands of networks. These actors are all boundedly rational and have varying levels of information on the state of their own and other's networks in the past and at this moment. To make things more complicated, their actions influence each other, can be mutually incompatible or mutually beneficial. They all have different assumptions, what caused their own actions and how this influenced the actions of others in the past and in the future. To completely mess things up, even if all data points and assumption were fully true until that point, that would not allow them to predict the future state of the network and routing of traffic (not even a second in the future!). It would be impossible to know whether their decision is "the most optimal", because routing is part of a chaotic system, called "the World". The state of the network and the traffic flowing through it, is not a closed system. It is influenced by everything that happens outside the network and that can be a lot, varying from war, natural disasters and pestilence to human error, Taylor Swift and people helping each other in times of need. That doesn't mean that the system is completely random, but it can't be relied on to be stable and it changes have to be expected and dealt with at any time. So the best routing system is the one that requires the least complete knowledge of the state of the network and can handle changes to that state at great speed and with great resilience.

The routing of traffic within one network is  determined by whoever is in control of that network. No two networks need to be the same, because the preferences and needs of those who control it may be different. Beautiful words can be spoken, about aligning the operation of the network and routing of traffic with lofty goals such as carbon awareness, customer needs etc. The reality is that there are many constraints and influences that will make the alignment of the daily operation of the network with those beautiful words and lofty goals a matter of opinion and not fact. There is therefore no most optimal way of designing a network or routing traffic over it. Each network is different and therefore operates differently. Interconnection of network makes it even more complex to operate a network optimally.

When it comes to cross domain routing concepts such as path dependent (SCION),  carbon based, cost based, QoS based routing were all tried in the 1980s and 1990s, failed and  surpassed by BGP4. All those earlier and later failed routing protocols made the same flawed assumption as carbon free routing does today; if I know the state of the network, I can choose the most optimal route for me and the whole world. In the late eighties most routing protocols required that the state of the network was known and couldn't change too much. This almost broke the Internet. Fortunately advances allowed BGP to be developed. It requires remarkably little knowledge of the state of other networks. It can't see the difference between an Autonomous System Number of BT vs JANET or Google. It doesn't know what routers they use, what bandwidth there is, what physical distance there is etc. It knows how to get from one ASN to another ASN. And despite not knowing what a router, capacity, a telco, content provider, customer, supplier, carbon, money or a country border is, it works rather well. It turns out all those other variables may be very important to individuals who operate one network, but aren't relevant to inter-domain routing. A basic explanation I wrote of the economic transactions that take place,can be found at. https://arstechnica.com/features/2008/09/peering-and-transit/ (This model is different and the opposite of the more often cited, but flawed, two-sided models of interconnection of Laffont, Marcus, Rey and Tirole)

Carbon aware routing in one network needs an accurate model of the electricity network
I'm quite sceptical of carbon aware routing within one network and even more so when it happens across networks. For it to work, the routing algorithm would need a full understanding of how the exhaust of carbon is the result of electricity generation. The paper that was presented does not have such a model as part of the CATE algorithm. The paper says that national and European electricity markets are interconnected, but the algorithm doesn't  derive the consequence from it: That the electricity network in Scotland is to a large extent the same one as in England and is interconnected with the electricity networks of EU countries. The total generation is related to the total consumption in the UK and that apart from losses in transport, import and export and  congestion, the total electricity consumption of the UK is relevant and not the location.

 Routing traffic through Scotland doesn't make the wind blow faster in Scotland. It doesn't use less electricity in the UK.  Most low carbon and zero carbon electricity sources aren't variable with demand. Their yield is driven by the intensity of the wind, water and the sun. Green electricity therefore has a priority over fossil fuel based electricity. Fossil fuel is the back-up for what can't be generated by renewable sources. (there are some renewable sources that may handle short peaks, eg water for TV-pick up in the UK) So shutting down an optic from London to Leeds may save electricity. This good, because it means a bit less electricity is needed in total and so a little bit less non-renewable fuel needs to be burnt. However, it will not lead to less carbon emitted, if shutting down a link to Leeds means that for traffic to Middlesborough a link from Manchester to Edinburgh and then to Middlesborough needs to be activated. The extra hop and longer distance for Edinburgh would ctually increase the total electricity consumption of the UK. This would require more fossil fuel to be burned. Scotland's green energy is a fixed quantity. Scotland doesn't become less green when traffic is routed through England. The UK does become less green if more electricity is needed in total to route traffic through Scotland. So a carbon aware routing protocol can only work if the model of the electricity network is complete and accurate for both how electricity is generated, consumed and distributed.

What if Scotland was independent from  England, Wales and Northern Ireland, would that be better?
Theoretically it could be possible that the electricity grid of Scotland is completely independent of England, Wales and Northern Ireland. In that case the electricity consumption and the carbon footprint would be independent metrics. Scotland would need to have an over supplied and 100% green grid,  completely independent of England, with no interdependence and no interconnection. If at the same time England has a network that still does emit CO2 in significant quantities to generate electricity, then there might be a form of arbitrage possible. In this case it might be possible that linking parts of BT's network in England, Wales and Northern Ireland through Scotland results in less carbon emissions by the combined electricity networks of Scotland, England, Wales and Northern Ireland.

Carbon aware routing in one network requires an accurate power profile of each device in the network
In the paper the authors assume that routers have a linear energy consumption profile above a base load. This is an academic oversimplification that is unusable in practice. Routers combine and split incoming data flows so that they become outgoing data flows. The complexity of a router is that incoming data flows and outgoing data flows are completely different from each other in size, distribution and compilation, though in aggregate they should be exactly the same. As a massive intersection of roads that should have no traffic jams or crashes. As a router reaches the maximum capacity of its ports and backplane its energy consumption is likely to increase in a non-linear fashion, heating it up more and requiring more cooling. At the same time it increasingly runs against its limits. Though the likes of Google have shown that they can get close to the technical limit of their networking equipment in the datacenter network for their services, it is very clear that this is a non-trivial task, that required a lot of upfront effort into knowing the behaviour of all their equipment and of the applications that run over it. https://cloud.google.com/blog/topics/systems/the-evolution-of-googles-jupiter-data-center-network

In the case of a telecom operator such as BT, the electricity consumption of the network is less deterministic. BT has a large number of very different types of users and usages. It has different agreements with those users. The equipment is at best of 2 different generations; the previous one and the one it is moving to. Its energy consumption is in part determined by physical distances, but also by weather! Certainly now climate change makes it warmer and sunnier in the UK too! The network changes every hour of every day, due to wear, tear, upgrades, downgrades, cable breaks, equipment failure etc. All of this isn't known to BT at any given moment and can take a while to percolate through to configuration databases at which point the state of the network is already different.

What the CATE algorithm tries to do therefore falls far short of what would be needed, because it makes linear assumptions and assumes a stable state of BTs network, while not correctly incorporating the operation of the electricity network.

Carbon aware routing across independent networks will break the Internet
Routing based on BGP makes use of Autonomous Systems and how many hops there are between the source ASN and the target ASN. Less hops is better. BGP doesn't make assumptions about what an ASN represents. BGP doesn't know anything about how many routers, how much capacity, how many miles or how much traffic is in each ASN. Between the more than 75K ASNs there are continuous changes and BGP will announce them and algorithms will process them. It is a dumb, but scalable system, with a remarkably efficient and scalable network of interconnected networks as an emergent property.

Carbon awareness would require that the routing algorithm actually knows and understands the electricity network that each piece of equipment in the entire Internet. Everything that should have been done in the BT network for the CATE algorithm to have reliable results, but then for every autonomous system and all the networks comprising those ASNs.

What would really break the Internet however is that if carbon is used as a metric to guide routing decisions, than certain paths will always emerge as the most optimal from a carbon perspective. This is clear from the lament of the authors about the Geant network relying so much on Germany for its connectivity and Germany having such high carbon emissions. If carbon aware routing was a thing on the Internet, all traffic would want to evade networks that run through Germany (or not if they actually understand that Europe has an interconnected electricity market). What would happen is that traffic would seek the route with the lowest carbon emissions and all traffic would be sent to that route, without consideration for capacity, costs etc. That route would be overloaded, of which it would have to alert other networks, but those wouldn't be able to determine which network can and which network shouldn't send its traffic over the saturated, but lowest carbon, link.

A similar issue was relayed to me by ISOC staff after the great earthquake in Japan, where most links had broken and therefore limited capacity was available. Japanese telecom firms had sold more "guaranteed" capacity than there was capacity available in the affected area. The result was that the routers wouldn't know what network had priority and therefore transmitted no traffic at all, despite total traffic being less than what little capacity was left. After the earthquake businesses were closed and most people were either in shelters hiding or doing practical things for the recovery and therefore the network was less in use.

Carbon, QoS and other such metrics are of little use in routing across autonomous systems. Each network is different and has its own operations. The success of the Internet is that BGP doesn't have to factor it all in.

Electricity aware networking not carbon aware routing
The paper does show a more practical approach limiting the negative impact of use of resources by networks. By looking at the electricity consumption of links and interfaces and the amount of traffic, there can be a reason to turn off certain interfaces and equipment during moments of low use. Even this is non-trivial, because of non-linear effects, redundancy and the ever changing state of the network and the performance of equipment. Looking at operational electricity consumption is probably difficult enough for network staff to deal with. Embedded carbon of new versus old equipment might be too complex already. To illustrate, a network may supply connectivity to a customer through 2 times 2x100Gbps on a redundant route between two towns. The peaks exceed 100Gbps on the link, but not by much and traffic growth can be limited. 2 times 400Gbps will then use less electricity at any moment. This can be calculated and demonstrated. The embedded carbon of earlier replacement of interfaces, combined with the network being able to use 2 ports for another customer are much more complex, particularly when the 100Gbps is used elsewhere in the network.

Carbon aware placement of datacenters
Transport losses, congestion and other examples of inefficiency and scarcity in real world electric networks might make it preferable to build sites that consume a lot of electricity close to where the generation is greenest. That however applies to BT's peers and not to BT itself. BT is a nationwide network. Its electricity consumption primarily follows where people live and work. That typically is England and not Scotland. So placing a government or hyperscale datacenter in Glasgow might be preferred over Slough from a carbon aware electricity generation point of view. Transmitting photons over optical fibre uses less electricity than the losses of the electricity network over the same distance.

Conclusion
The impact of networking on the planet is significant. Every attempt should be made to decrease the impact whether it is from the operation of the network or building, upgrading and replacing it. Using less electricity in networking is good, because it means that less electricity needs to be generated, whether it is through sustainable or unsustainable generation. Making routing decisions depend on assumed carbon emissions is however not going to lead to less energy consumption, less carbon emessions or better routing and networking. It will likely break the Internet and emit more carbon. Better awareness of electricity consumption and the lifetime of equipment is what the internet community can work on.