Hey everyone, Alex here. Welcome back to another edition of Coding with Alex on sysseder.com.
If you've been scanning the tech feeds this morning, you likely saw the headline: Chess.com Leak Exposes 7.3M Users, Evidence Points to Scraping. At first glance, a lot of non-technical folks see the word "leak" or "breach" and immediately assume some master hacker bypassed a firewall, ran a SQL injection, and dumped a database. But as developers, DevOps engineers, and security-minded architects, we know the reality is often far more subtle—and in many ways, more frustrating to defend against.
The evidence in the Chess.com incident points to automated data scraping. No cryptographic keys were stolen, no servers were compromised, and no databases were dropped. Instead, malicious actors simply used the platform's public-facing endpoints exactly how they were designed to be used—just at a massive, automated scale, systematically harvesting user profiles, emails, and match histories.
This raises a critical question for those of us building modern web applications: How do we protect our public data endpoints from being systematically harvested without breaking the user experience for legitimate browsers and mobile apps?
Today, we're going deep into the architecture of anti-scraping. We will move beyond basic rate limiting and look at modern, developer-first defenses—including TLS fingerprinting, behavioral analysis, and cryptographic proof-of-work challenges.
The Anatomy of a Scraping Attack
To defend against scraping, we first have to understand how modern scrapers operate. Gone are the days when a simple Python script using urllib or requests could easily harvest millions of records. Any basic Web Application Firewall (WAF) blocks those by looking at the User-Agent header.
Today's scrapers are highly sophisticated. They use headless browsers (like Playwright or Puppeteer) running in stealth mode, utilize residential proxy networks to rotate thousands of IP addresses, and mimic human behavior down to the millisecond. If your API returns public profile data, a scraper looks indistinguishable from a legitimate user navigating your frontend.
Let's look at the defensive layers we can implement in our application stack to stop them.
Layer 1: Advanced Rate Limiting with Token Buckets and Redis
Basic IP-based rate limiting is no longer sufficient because of residential proxy pools. However, it remains your first line of defense to stop low-sophistication, high-volume bots. Instead of a naive "100 requests per minute per IP" rule, we should implement a dynamic, sliding-window or token-bucket rate limiter that tracks multiple identifiers (such as session tokens, API keys, and IP subnets).
Here is a practical Node.js and Redis implementation of a sliding-window rate limiter designed to prevent aggressive endpoint harvesting:
const Redis = require('ioredis');
const redis = new Redis();
async function isRateLimited(identifier, limit, windowSizeInSeconds) {
const now = Date.now();
const clearBefore = now - (windowSizeInSeconds * 1000);
const key = `rate_limit:${identifier}`;
// Run as a transaction to avoid race conditions
const multi = redis.multi();
// Remove elements older than our window
multi.zremrangebyscore(key, 0, clearBefore);
// Add the current request timestamp
multi.zadd(key, now, now);
// Count total requests in this window
multi.zcard(key);
// Set a TTL so the key cleans up automatically if inactive
multi.expire(key, windowSizeInSeconds);
const results = await multi.exec();
const requestCount = results[2][1];
if (requestCount > limit) {
return true; // Rate limit exceeded
}
return false;
}
By identifying users not just by their IP, but by their authenticated session token or even JA3 TLS fingerprint, we can apply different limits. For instance, an unauthenticated user hitting /api/users/:username might have a tight limit of 10 requests per minute, whereas an authenticated user might have 200.
Layer 2: Fingerprinting Beyond User-Agents (JA3/JA4)
If a scraper rotates their IP address every three requests, our Redis rate limiter above will struggle to link them to a single actor. This is where TLS Fingerprinting (specifically JA3 and the newer JA4 specification) becomes a superpower.
When a client initiates a TLS handshake with your server, it sends a Client Hello message. This message contains supported cipher suites, TLS extensions, elliptic curves, and point formats. Because these parameters depend on the underlying cryptographic library (like OpenSSL, Go's crypto/tls, or NSS) rather than the HTTP headers, they create a highly unique "fingerprint."
- A legitimate Chrome browser on Windows has a specific JA3 fingerprint.
- A Python
requestsscript running on Linux has a completely different JA3 fingerprint. - Even a Puppeteer headless browser running on Node.js leaves a distinct cryptographic signature compared to a standard desktop browser.
By checking the JA3 fingerprint at your reverse proxy (like Nginx, HAProxy, or Cloudflare Workers), you can block or challenge requests that claim to be "Chrome" in their User-Agent header but present a Python-like TLS handshake. Here is a conceptual architecture of how this looks in your pipeline:
[Incoming Request]
│
▼
[Reverse Proxy / WAF] ──► Extracted JA3 Fingerprint: e7d705a2e...
│
├──► Match with User-Agent? No ──► [Block / CAPTCHA]
│
└──► Match with User-Agent? Yes ──► [Forward to Application API]
Layer 3: Cryptographic Proof-of-Work (PoW) Challenges
When an endpoint must be public (like a search or profile view) and rate limits are being bypassed via proxies, you can introduce a Proof-of-Work (PoW) challenge.
Instead of forcing a human to solve an annoying CAPTCHA (which ruins the UX and can be solved by automated AI solvers anyway), you force the browser to solve a quick cryptographic puzzle before the API yields any data. While a single human browser won't mind spending 100ms of CPU time to compute a hash, a scraper trying to harvest 7 million profiles will find their CPU resources instantly exhausted.
Here is how you can implement a basic PoW challenge in your API:
Step 1: The Server Issues a Challenge
When the client requests a sensitive endpoint, the server responds with a 402 Payment Required (or a custom 403 variation) containing a random salt and a difficulty target.
// Express.js middleware example
app.get('/api/profiles/:id', async (req, res, next) => {
const powVerified = verifyProofOfWork(req.headers['x-pow-nonce'], req.headers['x-pow-salt']);
if (!powVerified) {
const salt = crypto.randomBytes(16).toString('hex');
const difficulty = 4; // Number of leading zeros required in SHA-256 hash
return res.status(428).json({
error: "Precondition Required: Solve Proof of Work",
salt: salt,
difficulty: difficulty
});
}
// Proceed to serve the profile data...
});
Step 2: The Client Solves the Puzzle
In your frontend JavaScript, you catch this status code, run a fast hashing loop to find a nonce that, when combined with the salt, produces a SHA-256 hash with the specified number of leading zeros, and then retries the request.
async function solveChallenge(salt, difficulty) {
let nonce = 0;
const prefix = '0'.repeat(difficulty);
while (true) {
const data = salt + nonce;
const hash = await crypto.subtle.digest('SHA-256', new TextEncoder().encode(data));
const hashHex = Array.from(new Uint8Array(hash)).map(b => b.toString(16).padStart(2, '0')).join('');
if (hashHex.startsWith(prefix)) {
return nonce;
}
nonce++;
}
}
By dynamically adjusting the difficulty based on the client's risk score (determined by their JA3 fingerprint or IP reputation), you can surgically target suspicious scrapers without affecting your legitimate web visitors.
Layer 4: Data Obfuscation and Honeytokens
What if the scrapers still get through? If your data is displayed on a web page, they can scrape the HTML. Your final line of defense is making the harvested data useless or incredibly expensive to clean.
1. Dynamic Class Names and HTML Structure
Scrapers rely heavily on CSS selectors (e.g., document.querySelector('.user-email')). By using CSS Modules or Tailwind CSS with dynamic utility compilation during build time, you can randomize your HTML class names on every deployment. What was .user-email today becomes .css-x92f1b tomorrow, breaking the scraper's parsing selectors.
2. Honeytokens (Canary Accounts)
Seed your database with fake "honeytoken" profiles. These are accounts that no real human would ever visit naturally, but a systematic scraper harvesting the entire database from ID 1 to 10,000,000 will inevitably hit.
Configure your application logs to immediately trigger a high-severity PagerDuty alert the second a request hits a honeytoken endpoint. You can instantly block the source IP or proxy network before they steal the rest of your database.
Conclusion: Security is a Cat-and-Mouse Game
The Chess.com incident is a stark reminder that data security isn't just about protecting write access; it's also about protecting read access. As developers, we must design our APIs under the assumption that if public data can be viewed, someone will try to harvest it at scale.
By implementing a layered defense—combining multi-factor rate limiting, TLS fingerprinting, silent PoW challenges, and strategic honeytokens—you can make scraping your platform so computationally expensive and complex that bad actors will simply give up and move on to easier targets.
What about you? How are you securing your public-facing APIs? Have you experimented with TLS fingerprinting or proof-of-work challenges in production? Let me know in the comments below, or hit me up on Twitter/X at @sysseder!
Until next time, keep your dependencies updated and your endpoints rate-limited.
— Alex