We’ve all seen the flashy marketing copy from the latest LLM releases. "Our new model scores 92% on SWE-bench!" or "Achieves state-of-the-art coding capabilities!" It sounds incredible. But then you pull the API key, plug the model into your team's internal codebase, ask it to resolve a relatively straightforward Jira ticket, and... it hallucinates a library that doesn't exist, breaks your build, and tries to rewrite your database migration in a way that would make your DBA weep.
Why is there such a massive disconnect between benchmark scores and real-world developer experience?
The truth is, standard developer agent benchmarks like SWE-bench are starting to show their age. They are largely built on open-source repositories, many of which have leaked into the training data of these models (a phenomenon known as data contamination). Furthermore, open-source repos are structurally different from the massive, sprawling, slightly messy, and highly opinionated codebases we work on inside enterprise walls.
This is where Real-SWE comes in. It’s a new benchmarking framework designed to evaluate AI models on private, real-world, enterprise codebases. Today, we’re going to look under the hood of how we actually measure AI coding capabilities, why public benchmarks are failing us, and how you can think about benchmarking AI agents against your own private repos.
The Problem with the Status Quo: Why SWE-bench Isn’t Enough
To understand why Real-SWE is a big deal, we first need to look at what we've been relying on. SWE-bench (and its variants like SWE-bench Lite) has been the gold standard for evaluating AI software engineering agents. It takes a GitHub issue and a repository, and tasks the AI with generating a patch file that resolves the issue and passes the associated unit tests.
While revolutionary when it launched, SWE-bench suffers from three critical flaws that make it a poor proxy for enterprise utility:
- Data Contamination: The repositories used in SWE-bench (like Django, SymPy, and Scikit-learn) are incredibly popular. Their source code, issue trackers, pull requests, and unit tests have been scraped, parsed, and digested by frontier models during pre-training and reinforcement learning phases. The models aren't "solving" these problems from first principles; they are often just recalling the solution from memory.
- The "Cleanliness" Bias: Top-tier open-source projects have exceptionally clean architectures, strict style guides, and comprehensive test suites. Enterprise code, on the other hand, is a historical document. It contains legacy systems, undocumented design patterns, "temporary" hacks that became permanent, and patchy test coverage.
- Execution Environments: SWE-bench relies on static execution environments. In the enterprise, your code relies on internal APIs, microservices, private package registries (like JFrog Artifactory or private npm), and cloud-native infrastructure that cannot be easily spun up in a sandbox.
Enter Real-SWE: Benchmarking Behind the Firewall
Real-SWE addresses these issues head-on by shifting the benchmarking paradigm. Instead of evaluating models on public datasets, it provides a standardized framework to run evaluations inside secure, isolated enterprise environments using your actual, private codebases. It measures how an AI agent performs when it has no prior knowledge of the codebase, forcing it to rely entirely on its reasoning, repository-navigation, and execution capabilities.
Let's look at the core architectural difference between how traditional benchmarking works versus the Real-SWE approach:
+-----------------------------------------------------------------------+ | TRADITIONAL BENCHMARKING (e.g., SWE-bench) | | | | [Public GitHub Repo] ----> [Model Training Data] (Contamination!) | | | | | [Static Issue Evaluation] --------> [LLM Agent] ---> [Static Patch] | +-----------------------------------------------------------------------+ +-----------------------------------------------------------------------+ | REAL-SWE BENCHMARKING FLOW | | | | [Private Enterprise Repo] (Zero LLM Exposure) | | | | | v | | [Secure Agent Sandbox] <---> [Real-SWE Orchestrator] | | | | | | v v | | [Dynamic Execution Environment] <-> [LLM / Coding Agent] | | (Private APIs, DBs, Custom Tests) | +-----------------------------------------------------------------------+
By executing agents inside a controlled, dynamic environment that mirrors your actual staging or development setup, Real-SWE evaluates the agent's ability to truly "read" code, trace dependencies, write patches, and iterate based on actual compiler and test execution feedback.
How to Design an Internal AI Benchmark
If you're looking to evaluate code generation models (like GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro) or developer tools (like Cursor, Devin, or custom LangChain/AutoGPT setups) for your team, you don't have to wait for a vendor tool. You can implement the core principles of Real-SWE yourself.
An effective internal benchmark requires three components: a curated set of evaluation tasks, an execution sandbox, and a validation runner.
Step 1: Selecting the Evaluation Tasks
Do not use synthetic tasks. Instead, go back through your git history and pick 10-15 historical pull requests that met the following criteria:
- They resolved a clearly defined bug or implemented a small, self-contained feature.
- They touched multiple files (to test the agent’s repository navigation).
- They had associated unit or integration tests that failed before the fix and passed after.
Step 2: Defining the Task Schema
For each task, you need to create a JSON definition file that outlines the starting commit, the target goal (the issue description), and the validation commands. Here is an example of what a task definition might look like for an internal Node.js/Express service:
{
"task_id": "TASK-1042-billing-rounding-error",
"repository": "git@github.com:our-org/billing-service.git",
"base_commit": "8f3a1d9c72e4b50d61a8c7e93f5a1b2c3d4e5f6a",
"problem_description": "We are seeing rounding errors on multi-item invoices when tax is applied per-item instead of to the subtotal. Update the invoicing logic in src/services/tax.ts to calculate tax on the final subtotal rather than accumulating rounded per-item tax values. Ensure all existing billing tests still pass.",
"setup_commands": [
"npm install",
"npm run db:migrate:test"
],
"validation_commands": [
"npm run test:unit src/tests/tax.test.ts"
],
"timeout_seconds": 300
}Step 3: Implementing the Execution Sandbox
You must never let an untrusted AI agent run arbitrary bash commands directly on your local machine or a shared development server. An agent trying to fix a dependency issue could easily run a destructive command or expose environment secrets.
We use Docker to create an isolated, reproducible sandbox for the agent. Below is a simplified Python orchestrator script that demonstrates how to check out a specific commit of your codebase, pass the task description to your agent, and run the validation tests within a secure Docker container.
import docker
import os
import git
def run_evaluation_task(task_config):
client = docker.from_env()
repo_path = "/tmp/eval-repo"
# 1. Clone and checkout the base commit in a temporary directory
if os.path.exists(repo_path):
os.system(f"rm -rf {repo_path}")
repo = git.Repo.clone_from(task_config["repository"], repo_path)
repo.git.checkout(task_config["base_commit"])
# 2. Spin up the isolated Docker container mounting the repo
print(f"Starting sandbox for {task_config['task_id']}...")
container = client.containers.run(
image="node:18-alpine",
command="/bin/sh -c 'sleep 3600'", # Keep container alive
volumes={os.path.abspath(repo_path): {'bind': '/app', 'mode': 'rw'}},
working_dir="/app",
detach=True,
network_mode="bridge" # Limit access if security is a concern
)
try:
# 3. Run setup commands inside the container
for cmd in task_config["setup_commands"]:
print(f"Running setup: {cmd}")
container.exec_run(cmd)
# 4. Hand off to your AI Agent (Conceptual)
# Here, your agent would read /app, modify files based on the problem_description
print("Invoking AI Agent to resolve the issue...")
# agent.resolve_issue(repo_path, task_config["problem_description"])
# 5. Run validation tests to verify the fix
print("Running validation tests...")
success = True
for val_cmd in task_config["validation_commands"]:
result = container.exec_run(val_cmd)
print(result.output.decode('utf-8'))
if result.exit_code != 0:
success = False
break
if success:
print(f"Result: SUCCESS! The agent resolved {task_config['task_id']}")
else:
print(f"Result: FAILED! The agent's patch did not pass validation tests.")
finally:
# Always clean up the container
print("Cleaning up sandbox container...")
container.stop()
container.remove()
# Example invocation:
# run_evaluation_task(task_config)What Real-SWE Teaches Us About LLM Evaluation
When teams begin running internal benchmarks like this, they quickly realize that raw "coding capability" (i.e., writing syntactically correct code blocks) is rarely the bottleneck for AI agents. Instead, the failure modes are almost always related to:
- Context Window Management: Can the agent find the correct 3 files out of a repository containing 2,000 files without blowing past its context token limit?
- Dependency Resolution: Does the agent understand our internal lockfiles? Can it handle private scope npm packages or internal Maven repositories?
- Feedback Loops: When a unit test fails, can the agent read the stack trace, understand the assertion error, and modify its code, or does it get stuck in an infinite loop repeating the same mistake?
These are the core metrics that Real-SWE surfaces. It forces us to stop treating LLMs like isolated calculators and start treating them like real developers operating in complex environments.
Wrapping Up: Build Your Own Benchmarks
If your organization is serious about integrating AI into its software engineering workflows, stop relying on generic vendor benchmarks. A model scoring 90% on SWE-bench might score 15% on your internal, highly-custom microservice architecture.
By building a lightweight, dockerized evaluation harness modeled after the Real-SWE philosophy, you can safely, accurately, and quantitatively find out which LLM actually works for your codebase. It prevents vendor lock-in, stops you from paying for over-hyped models, and ensures that when you do roll out AI tooling to your engineering team, it actually boosts productivity instead of developer frustration.
Over to you: Have you tried running AI coding agents against your company’s private repos? What was the biggest blocker you ran into? Let’s talk about sandbox environments and context retrieval strategies in the comments below!