I remember us having an extended discussion about that data internally back in October. We read the methodology, determined that the error bars on Ecoping's CO₂e estimates were too wide for a meaningful comparison between search engines, and swapped the table out for a new one that only reports metrics we can measure with confidence.
Looks like there were two copies of the table and I only replaced one of them. I'll go fix that.
For those interested, relevant bits of the internal discussion:
I'm reading through the way the service we used for those calculations works. I want to say two things simultaneously:
- A lot of good thought has gone into how those numbers were calculated. Like, a lot more than I anticipated, their methodology section is pretty good. (At least when you click through to latest version of the spec, I'm less sure about the specific implementation we used.)
- Estimating CO₂ emissions of a website by making observations about it from the outside (approximately bytes transferred + grid CO₂ intensity based on server IP range) leaves you with error bars too large for this to be a meaningful comparison between search engines. There's just not enough data going into the formula they're using to get a specific enough answer out.
But imagine that I open assistant and have Claude Opus write me a short story 10k words long, and then while it's working I go to https://kagi.com/news to catch up on what's happening in the world. The formula we're using would say that loading the news page had twice the ecological impact as querying the LLM [because images use more bandwidth than text]. That's crazy.
[I]t'd be best to remove [...] CO₂ from that table and stick to bytes transferred and page load time, because we can get good measurements of those things. I'd love to know CO₂e/search myself, but getting a number we could actually trust would involve looking inside every cloud service and API we use, and I doubt vendors will play along with that.
