Hey everyone, Alex here. Welcome back to another edition of Coding with Alex at sysseder.com.
If you've been glancing at the tech headlines today, you might have seen a monumental milestone: solar energy has generated more power in the US than coal for the first time in history. On the surface, this feels like a win for environmentalists and utility companies. But as software engineers, site reliability engineers (SREs), and cloud architects, why should we care?
The transition from fossil-fuel baseload power to renewable energy sources like solar and wind represents a paradigm shift. Unlike a coal or gas plant that you can spin up or down on demand, solar power is intermittent and variable. It peaks mid-day and drops to zero at night. Because our cloud workloads run on massive, power-hungry data centers that draw from these exact same grids, the way we design, deploy, and run software has to evolve.
Today, we're diving into the engineering discipline of Green Ops (or carbon-aware software engineering). We'll explore how to architect applications that adapt to a solar-powered grid, run through some concrete architectural patterns, and write some Python code to shift our batch processing to match peak solar generation.
The Green Ops Paradigm: Shifting from High-Availability to Carbon-Intensity
For decades, cloud architecture has been driven by three main pillars: performance, availability, and cost. If a server is running hot or an API response time spikes, we throw more compute at it. Autoscale groups spin up new VMs, and Kubernetes schedules more pods.
However, when the grid is increasingly powered by solar, there is a direct correlation between when you run your compute and the carbon footprint of your application. Running a heavy database migration at 8 PM (when solar is gone and gas-fired peaker plants are running) has a vastly higher carbon intensity than running it at 1 PM (when there is an excess of solar energy on the grid).
As developers, we have two primary levers to pull to make our applications carbon-aware:
- Temporal Shifting: Moving workloads in time to coincide with periods of high renewable (solar) generation.
- Spatial Shifting: Moving workloads in space (geographically) to data centers located in regions where the sun is currently shining and solar energy is abundant.
How to Measure Grid Carbon Intensity in Real-Time
To write carbon-aware code, we first need data. We can't shift our workloads without knowing when the grid is "clean" (high solar/renewables) versus "dirty" (high fossil fuels).
Thankfully, there are fantastic APIs available for this. The industry standard is Electricity Maps, and another great free tool is the WattTime API. These APIs provide real-time carbon intensity metrics measured in grams of CO2 equivalent per kilowatt-hour (gCO2eq/kWh) and forecast data for the next 24 to 72 hours.
Let's look at a conceptual architectural flow of how a carbon-aware application operates:
+-------------------+ Query Forecast +------------------------+
| Our Scheduler | -------------------------> | Grid Carbon API |
| (Cron / Worker) | <------------------------- | (e.g., ElectricityMap)|
+-------------------+ JSON Forecast Data +------------------------+
|
| (Analyze data: identify lowest carbon intensity window)
v
+------------------------+
| Execute Workload | -> Runs during peak solar hours!
| (Kubernetes Job, etc.) |
+------------------------+
Building a Carbon-Aware Task Scheduler in Python
Let's write a practical Python script. Imagine you run a SaaS platform that generates heavy PDF reports for users or processes massive video transcodes. These tasks are not time-critical; they just need to be done within a 12-hour window.
Instead of running them immediately or on a static midnight cron job, we can query a carbon intensity API, find the optimal "solar window," and schedule our background worker to execute then.
First, let's install our HTTP client dependency:
pip install requests pandasNow, let's write a carbon-aware scheduler. For this example, we will mock a carbon forecast response that represents a typical 24-hour cycle where solar production peaks during the afternoon (dropping carbon intensity significantly).
import datetime
import requests
import json
# Mocking the API response for demonstration.
# In production, replace this with a call to Electricity Maps or WattTime API.
def get_carbon_intensity_forecast(zone="US-CA"):
"""
Returns a list of forecasted carbon intensities (gCO2eq/kWh) for the next 12 hours.
Lower values mean cleaner energy (high solar/renewables).
"""
now = datetime.datetime.utcnow()
forecast = []
# Let's simulate a curve where carbon intensity drops in the afternoon due to solar abundance
for hour_offset in range(12):
target_time = now + datetime.timedelta(hours=hour_offset)
hour = target_time.hour
# Simple curve: lowest intensity (cleanest) between 11 AM (11:00) and 3 PM (15:00)
if 11 <= hour <= 15:
intensity = 120 # Very clean, high solar generation
elif 8 <= hour < 11 or 15 < hour <= 18:
intensity = 250 # Moderate intensity
else:
intensity = 450 # High intensity, fossil fuels dominate the grid at night
forecast.append({
"datetime": target_time.isoformat(),
"carbon_intensity": intensity
})
return forecast
def find_optimal_execution_time(forecast, execution_duration_hours=2):
"""
Analyzes the forecast and finds the consecutive window with the lowest average carbon intensity.
"""
best_start_index = 0
lowest_average_intensity = float('inf')
# Slide a window across the forecast array
for i in range(len(forecast) - execution_duration_hours + 1):
window = forecast[i : i + execution_duration_hours]
avg_intensity = sum(item["carbon_intensity"] for item in window) / execution_duration_hours
if avg_intensity < lowest_average_intensity:
lowest_average_intensity = avg_intensity
best_start_index = i
optimal_time = forecast[best_start_index]["datetime"]
return optimal_time, lowest_average_intensity
if __name__ == "__main__":
print("Fetching grid carbon intensity forecast...")
forecast_data = get_carbon_intensity_forecast(zone="US-CA")
# We have a video processing job that takes approximately 2 hours to run
job_duration = 2
optimal_start, expected_intensity = find_optimal_execution_time(forecast_data, job_duration)
print(f"\n--- Optimal Workload Schedule ---")
print(f"Target Zone: US-CA")
print(f"Job Duration: {job_duration} hours")
print(f"Best Time to Start: {optimal_start}")
print(f"Expected Average Carbon Intensity: {expected_intensity} gCO2eq/kWh")
print(f"---------------------------------")
By executing this logic within a custom Airflow operator, a temporal task scheduler, or even a serverless function, we transition our infrastructure from being passive consumers of energy to active, intelligent participants in the grid's stability.
Spatial Shifting: Following the Sun
While temporal shifting is great for delay-tolerant batch jobs, what about user-facing, low-latency APIs? You can't delay a user's checkout request until tomorrow afternoon just because the sun went down.
For high-availability, real-time web applications, we can leverage spatial shifting (also known as follow-the-sun routing). If you run multi-region Kubernetes clusters on AWS, GCP, or Azure, you can route non-stateful microservices, API read operations, or reporting queries to regions currently experiencing peak solar production.
Multi-Region Routing with DNS / Anycast
Using CDNs or Intelligent DNS routing (such as Cloudflare Workers or AWS Route 53 Geoproximity routing), you can dynamically adjust traffic weights based on the carbon footprint of regional datacenters.
For example, if it's 2 PM in Europe, European datacenters (like `eu-west-1` in Ireland or `eu-central-1` in Frankfurt) may have access to cleaner, solar-heavy grids. As the sun sets in Europe and rises in the US, routing metrics can dynamically shift compute workloads across the Atlantic to `us-east-1` or `us-west-2`.
Infrastructure Level Optimization: Kubernetes KEDA and Carbon-Aware Autoscaling
As DevOps engineers, we don't always want to write custom application code to handle carbon metrics. We want our infrastructure to do it natively.
This is where projects like the Kubernetes Event-driven Autoscaling (KEDA) combined with carbon-aware operators come into play. There are emerging open-source integrations (such as the Carbon-Aware KEDA Operator) that allow you to scale your pods up or down based on grid carbon intensity signals.
For example, you could define a Horizontal Pod Autoscaler (HPA) behavior that scales your background workers like this:
- Carbon Intensity < 150 gCO2eq/kWh (Peak Solar): Set `maxReplicas` to 50 (process the backlog at maximum throughput).
- Carbon Intensity > 350 gCO2eq/kWh (Nighttime/Gas): Set `maxReplicas` to 2 (scale down to minimum viable throughput to preserve grid power).
Example KEDA Scaler Manifest Concept
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: carbon-aware-worker-scaler
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: background-worker
minReplicaCount: 2
maxReplicaCount: 50
triggers:
- type: prometheus
metadata:
# Querying custom Prometheus metric scraped from Carbon Intensity APIs
serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
metricName: grid_carbon_intensity_gco2
query: sum(grid_carbon_intensity_gco2{region="us-west-2"})
threshold: '200' # Scale down/restrict if intensity goes above this
The Developer's Role in a Renewable Future
As the headline reminds us, the energy grid is transforming at an unbelievable pace. Coal is dying, and solar is taking the crown. But the cloud isn't just a virtual space—it's anchored by physical servers consuming gigawatts of electricity.
By writing carbon-aware applications, implementing temporal/spatial shifting, and building Green Ops metrics into our CI/CD pipelines and infrastructure autoscalers, we can directly reduce the carbon footprint of our digital systems. It's a fantastic intersection of software architecture, data-driven engineering, and environmental stewardship.
Over to you: Have you experimented with carbon-aware APIs or Green Ops tools in your current DevOps pipelines? Let me know your thoughts in the comments below, and don't forget to subscribe to the newsletter for more technical deep-dives!
Until next time, keep coding clean.