We’ve all been there: you’ve spent weeks optimizing your backend, shaving milliseconds off your database queries, and tuning your React rendering. Yet, when you look at your Real User Monitoring (RUM) metrics, your "Time to First Byte" (TTFB) still shows a frustrating spike for users on certain networks.
Often, the culprit isn't your code at all—it's the invisible dance of the TLS handshake. Specifically, a silent performance killer known as the HelloRetryRequest (HRR).
Recently, Cloudflare’s engineering team published a fascinating technical deep dive into how their Adaptive Key Exchange (AKE) pipeline slashed origin HelloRetryRequests from a staggering 52% to just 3.7%. If you are a DevOps engineer, cloud architect, or full-stack developer running traffic through reverse proxies, CDNs, or load balancers, this is a masterclass in modern network engineering.
Today, we are going under the hood of TLS 1.3, looking at why key exchange mismatches happen, and exploring how we can apply these architectural lessons to our own systems.
Understanding the TLS 1.3 Handshake (and the HRR Penalty)
To understand why this optimization is such a big deal, we need to quickly recap how TLS 1.3 streamlined the secure connection process.
In older versions of TLS (like 1.2), establishing a secure connection required a multi-step negotiation. The client and server had to agree on a cipher suite, then agree on a key exchange algorithm, and only then could they generate keys. This took two round trips (2-RTT) before any application data (like your HTML or JSON) could be sent.
TLS 1.3 changed the game by introducing a 1-RTT handshake. It achieves this through "optimistic" key generation.
The Happy Path (1-RTT)
In a standard TLS 1.3 handshake, the client doesn't just say "Hello." It makes an educated guess about which key exchange algorithms (groups) the server supports (for example, X25519 or P-256). The client generates its public key share for that specific group and sends it immediately in the initial ClientHello message.
Client Server
| |
| ---- ClientHello -----------------------> |
| + KeyShare (e.g., X25519) | [Server agrees on X25519]
| | [Generates Server Key Share]
| <--- ServerHello ------------------------ |
| + KeyShare (X25519) |
| + EncryptedExtensions, Certificate |
| + Finished |
| |
| ---- [Encrypted Application Data] ------> | (1 Round Trip Time!)
If the server supports the group the client guessed, it processes the share, sends its own key share back in the ServerHello, and boom—we have an encrypted tunnel in just one round trip.
The Unhappy Path: The HelloRetryRequest (2-RTT Penalty)
But what happens if the client guesses wrong?
Say the client sends a key share for P-256, but the server is configured to strictly require X25519 or a newer post-quantum algorithm like X25519Kyber768Draft00.
The server cannot read the client's key share. Instead of establishing a connection, the server must send a HelloRetryRequest (HRR). This message essentially says: "Nice try, but I don't support that group. Please try again using this specific group instead."
Client Server
| |
| ---- ClientHello -----------------------> |
| + KeyShare (P-256) | [Server says: "Nope, I need X25519"]
| |
| <--- HelloRetryRequest (HRR) ------------ | <-- EXTRA ROUND TRIP PENALTY
| (Please use X25519) |
| |
| ---- ClientHello -----------------------> |
| + KeyShare (X25519) |
| |
| <--- ServerHello ------------------------ |
| + KeyShare (X25519) |
| | (2 Round Trip Times...)
This adds an entire extra round trip (1-RTT penalty) to the handshake. On mobile connections or high-latency satellite networks, this extra round trip can add 100ms to 500ms of pure delay before a single byte of your website loads.
The Origin Conundrum: Why Was the HRR Rate at 52%?
You might expect HRR rates to be low. After all, modern browsers and major CDNs are highly compatible. However, Cloudflare discovered a massive performance bottleneck on the egress side of their network—the connections from Cloudflare's global edge proxies to customer origin servers.
When Cloudflare acts as a reverse proxy, it behaves as a TLS client when talking to your origin server. Because Cloudflare handles millions of diverse origins, configuring a one-size-fits-all key share guess is incredibly difficult.
If Cloudflare guessed X25519, but a legacy enterprise origin server only supported P-256, a HelloRetryRequest occurred. If Cloudflare guessed P-256, but a modern, hardened origin only accepted X25519 or post-quantum keys, an HRR occurred.
Because of this fragmentation across the web's infrastructure, over 52% of connections from Cloudflare to origin servers were suffering from the HRR penalty. Half of all origin fetches were taking twice as long to establish secure connections as they theoretically should have!
The Solution: Adaptive Key Exchange (AKE)
To solve this, Cloudflare built Adaptive Key Exchange (AKE). Instead of statically guessing a key group or sending multiple expensive key shares in every single request (which increases packet sizes and risks IP fragmentation), AKE turns the TLS handshake choice into a dynamic, learning system.
The architecture of AKE relies on three core pillars:
- In-Memory Connection Caching: Storing the successful key group of the last successful handshake for any given origin.
- Graceful Fallbacks: Instantly reacting to changes in origin configurations without dropping connections.
- Distributed State sharing: Ensuring that different edge servers within a data center can benefit from the learned configuration.
How AKE Works in Practice
Let's look at a conceptual implementation of how an Adaptive Key Exchange mechanism decides which key share to send. Rather than using static configurations, the proxy consults a fast, in-memory cache (like Redis or a localized shared-memory table) before initiating the TLS connection.
// Conceptual representation of an Adaptive Key Exchange decision engine
package main
import (
"context"
"fmt"
"sync"
)
type KeyGroup string
const (
GroupX25519 KeyGroup = "X25519"
GroupP256 KeyGroup = "P-256"
GroupP384 KeyGroup = "P-384"
DefaultGroup KeyGroup = GroupX25519
)
// AKECache tracks the preferred key group for each origin
type AKECache struct {
mu sync.RWMutex
store map[string]KeyGroup
}
func (c *AKECache) GetPreferredGroup(origin string) KeyGroup {
c.mu.RLock()
defer c.mu.RUnlock()
if group, exists := c.store[origin]; exists {
return group
}
return DefaultGroup // Fallback to safe default
}
func (c *AKECache) UpdatePreferredGroup(origin string, group KeyGroup) {
c.mu.Lock()
defer c.mu.Unlock()
c.store[origin] = group
}
// SimulateHandshake attempts a TLS handshake
func SimulateHandshake(origin string, cache *AKECache) {
guessedGroup := cache.GetPreferredGroup(origin)
fmt.Printf("[Client] Initiating handshake to %s guessing group: %s\n", origin, guessedGroup)
// Simulated Origin Server configuration
actualOriginSupportedGroup := GroupP256
if guessedGroup != actualOriginSupportedGroup {
// HRR Event occurs!
fmt.Printf("[Server] HelloRetryRequest! I do not support %s. Please use %s.\n", guessedGroup, actualOriginSupportedGroup)
// Update our adaptive cache for the next connection
cache.UpdatePreferredGroup(origin, actualOriginSupportedGroup)
fmt.Printf("[Client] Cache updated for %s. Subsequent connections will use %s.\n", origin, actualOriginSupportedGroup)
} else {
fmt.Println("[Handshake] Success! 1-RTT Connection established immediately.")
}
}
func main() {
cache := &AKECache{store: make(map[string]KeyGroup)}
originHost := "origin-legacy.example.com"
fmt.Println("--- FIRST CONNECTION ATTEMPT ---")
SimulateHandshake(originHost, cache)
fmt.Println("\n--- SECOND CONNECTION ATTEMPT ---")
SimulateHandshake(originHost, cache)
}
When you run this logic, the first request experiences an HRR but immediately trains the system. The second request hits the 1-RTT happy path perfectly. By caching this state, Cloudflare managed to drop their origin HRR rate from 52% to 3.7% globally.
What Developers and DevOps Engineers Can Learn From This
While most of us aren't operating global CDNs at Cloudflare's scale, this engineering achievement offers several critical takeaways for our own infrastructure design:
1. Audit Your Internal Services for HRRs
If you run a microservices architecture behind an API Gateway (like Kong, Traefik, or AWS ALB), your internal services are likely communicating via TLS. If your API gateway is misconfigured relative to your internal microservices, you could be paying the 1-RTT HRR penalty on every single internal API hop.
To inspect this, check your load balancer and proxy metrics for TLS handshakes. Many proxies allow logging of TLS handshakes that result in an HRR. If yours does, make sure your proxy's client configuration matches the preferred cipher and key exchange groups of your backend services.
2. Keep Caches Local and Fast
Notice that Cloudflare’s solution relies on extremely fast, low-latency lookups. If your connection-negotiation engine has to query a slow, remote database to find out which key group to use, the database query latency will completely negate the performance benefits of saving a TLS round trip! If you implement adaptive state tracking, always keep that state in local, thread-safe memory or high-speed local caches.
3. Align on Modern Standards
The simplest way to avoid HelloRetryRequests entirely is uniformity. Wherever possible, standardize your company's servers and containers on modern, highly-performant defaults like X25519 for key exchange. When your entire internal fleet speaks the same cryptographic dialect, the need for negotiation disappears.
Conclusion
Performance optimization is rarely about one single silver bullet. More often, it’s about identifying the friction points hidden deep inside our network protocols. By building Adaptive Key Exchange, Cloudflare eliminated a massive source of latency for millions of origin servers across the globe, proving that even fundamental protocol designs can be optimized with smart, adaptive software engineering.
Have you audited your own system’s TLS handshake metrics lately? Are you seeing unexpected latency spikes in your internal service mesh? Let’s talk about it in the comments below!
Looking to optimize your cloud infrastructure? Don't forget to subscribe to "Coding with Alex" for weekly breakdowns of networking, security, and cloud architecture deep dives!