Can AI do real engineering? Reading RWE-bench

On October 27, 2026, engineering-AI startup Xplore released RWE-bench (Real World Engineering Bench), which tests frontier models on 20 mechanical, electrical and aerospace engineering tasks. Every result is graded automatically by a deterministic physics verifier. Of the 11 launch models, the best, GPT-6 Astra, reached an average pass rate of 37%, and five models never passed a single run.

What it tests

Xplore's official page lists three task groups:

• Mechanical engineering, 9 tasks: parts and mechanisms modelled in CAD and checked for strength, fit and motion, including engine parts such as crankshafts, sheet-metal parts designed for manufacture, and 3D-printed mechanisms.
• Electrical engineering, 5 tasks: high-speed PCB routing on dense, already populated boards, including DDR memory and PCIe links, with length-matching, skew and impedance limits.
• Aerospace engineering, 6 tasks: CFD (computational fluid dynamics) and conjugate heat transfer, from building the mesh to a converged solution. Subjects include turbine vanes with film cooling, aircraft wings, rocket nozzles and cooling channels.

Xplore published three example tasks:

• Crankshaft design: design a one-piece steel crankshaft with oil drillings for a 3-cylinder diesel engine in CAD. It must fit the given bearings, flywheel and front drive, and clear the connecting rods as they turn. The counterweights must balance the shaft while staying under the mass limit. Under full-load firing at 2,400 rpm, stress, stiffness and bearing loads must stay within limits, and the shaft must survive millions of cycles without fatigue failure.
• DDR3 routing: route the DDR3L memory interface on a production FPGA board. It is a 32-bit fly-by bus from a Zynq-7000 to 2 memory chips, with 71 nets out of a 484-ball BGA on an 8-layer board. About 300 parts are already placed and every other net is already routed, and none of them may move. Data, address and command lines must each meet their skew budgets, and every track must hold its target impedance.
• Turbine vane CFD: predict the heat transfer around a high-pressure turbine vane. The agent must build and mesh the flow passage around the vane, run the CFD to a converged solution, and report the heat-transfer coefficient along the vane surface and the total heat load.

How it is graded

According to Xplore, each task was written by an engineer on its team with expertise in the relevant domain. The same engineer also wrote the verifier and built a reference solution that meets every requirement, using the same tools the agents get.

• The agent receives a technical brief and works in a sandbox with CAD/CAE tools for up to 24 hours per attempt. It can run its own validation before submitting.
• After submission, a deterministic program checks every requirement, running physics solvers where needed. A design passes only if it meets all of them. There is no LLM judge and no human grading.
• Each model runs in its vendor's native agent harness (for example Codex, Claude Code or Gemini CLI) at maximum reasoning effort, with 5 runs per model per task.

Results

The figures below are from the official page as of October 28, 2026. Pass rates are single-attempt averages, with 95% credible intervals in brackets. Cost is the mean API spend per run.

Figure 1: single-attempt average pass rate with 95% credible intervals. Data from Xplore's official page; chart by CADFrontier.

Model	Harness	Average pass rate	Cost per run
GPT-6 Astra	Codex	37% (30.7–43.8%)	$58.83
Claude Opus 5.5	Claude Code	29% (21.9–37.6%)	$57.52
GPT-6.1 Sol	Codex	25% (18.8–31.9%)	$10.28
Gemini 4 Argon	Gemini CLI	12% (7.5–18.5%)	$23.66
Claude Fable 5.1	Claude Code	7% (3.7–12.2%)	$97.57
Kimi K3.1 Pro	Kimi Code	1% (0.1–4.6%)	$71.20
DeepSeek V4.1 Flash	DSH	0% (0–2.1%)	$1.03
Qwen3.8 Max	Qwen Coder	0% (0–2.1%)	$17.40
GLM-5.3	ZCode	0% (0–2.2%)	$22.66
Muse Wonder	Muse Code	0% (0–2.1%)	$27.56
Grok 4.7	Grok Build	0% (0–2.1%)	$66.27

The official page also reports:

• When a task counts as passed if any of 5 attempts succeeds (pass@5), Claude Opus 5.5 passes 70% of tasks.
• Across all 1,100 runs, 4 tasks were never passed by any model.
• The average run took 3.4 hours, and longer runs did not score better.
• Cost does not track results either: GPT-6.1 Sol costs about $10 a run and scores only 4 to 12 points below the top two, which cost almost six times as much.

Figure 2: mean API cost per run against average pass rate. Data from Xplore's official page; chart by CADFrontier.

What it means for CAD engineers

The CAD here is code-based modelling, not a mainstream MCAD feature tree. The tools listed for the crankshaft example are CadQuery, Gmsh, CalculiX and Python: open-source, scriptable tools for modelling, meshing and finite-element analysis, not interactive parametric software such as SolidWorks, NX or Creo. The page does not say which tools the electrical and aerospace tasks use. A pass therefore means the design satisfies the physics checks, not that the model can deliver an editable native feature history in everyday CAD software. For more on that difference, see our article AI in CAD: the real change is in the feature tree.

Capable, but not yet consistent. The best single-attempt pass rate is 37%, yet Claude Opus 5.5 passes 70% of tasks within 5 attempts. Models can sometimes close a full engineering loop, but a single result is not reliable. Real-world use still needs repeatable automated checks and review by an engineer. For related discussion, see Reliable CAD agents: why every step needs verification.

The score covers the model and its harness together. Each model ran in its own vendor's agent harness, so the results reflect both the model and its toolchain. They should not be read as a ranking of the models alone.

Limitations

• Small sample: there are only 20 tasks, and the intervals are wide. The top two intervals overlap (30.7–43.8% and 21.9–37.6%), so the order of first and second place is not conclusive.
• Not independently reproducible: the tasks and verifiers are not public. Outside parties can only take part by contacting Xplore.
• Commercial interest: Xplore also offers reinforcement-learning training environments, so the benchmark is directly tied to its business. See the conclusion below.
• Results may change: the table reflects the official page as of October 28, 2026. Model names are given as Xplore lists them.

Conclusion: what the benchmark sets out to prove

According to Xplore, RWE-bench has two aims.

First, to measure the gap. AI now beats most people at math and coding, yet it still cannot reliably solve engineering problems that require balancing geometry, stress, heat and impedance at once. A top pass rate of 37%, and five models at 0%, put a measurable number on that gap.

Second, to show that engineering can be trained like code and math. In its thesis, Xplore argues that models advanced quickly in code and math because results there can be verified automatically, which lets models practise at scale through reinforcement learning with verifiable rewards (RLVR). Engineering results can be verified too: a turbine blade either stays within its temperature and stress limits or it does not. In Xplore's view, what is missing is data. Real engineering problems, real tools and checks that cannot be gamed mostly sit inside companies and in the experience of senior engineers, not in public repositories as code did. Because RWE-bench is graded only by deterministic programs and physics solvers, it is itself a demonstration of that verifiable approach.

Those two aims also define its commercial role. The official page offers two services: evaluating models for AI developers, and building reinforcement-learning training environments with the same approach (the benchmark tasks themselves are never licensed for training). RWE-bench is therefore both a measuring tool and a showcase for Xplore's training-environment business: first show that the gap is real and can be measured precisely, then supply the data needed to close it.

For engineers, the takeaway is this: AI can already run the full loop of modelling, meshing, solving and checking on some real engineering tasks, but it succeeds on fewer than four in ten attempts, using a code-based, open-source toolchain. On current evidence it is closer to an assistant that needs an engineer's oversight than to something that can deliver a design on its own.