Eight frontier models met a real codebase and the best one finished 39 percent of the work

😁 Hello, super humans! Every coding benchmark you have seen this year was built on public repositories, which means the model quite possibly read the answer during training. Specific Labs went and licensed ten private production codebases instead: a fintech platform chewing through bank statements, an events app with 200,000 users, a few enterprise sales tools. Then they handed eight frontier agents the actual tickets those engineers had been assigned. The leaderboard looks nothing like the one you are used to.

📰 Quick Signals

• 🧠 AI: Satya Nadella said Microsoft welcomes the “deliberate pacing” Dario Amodei called for and is publishing a Code of Conduct for its first-party MAI models today, opening it to public consultation rather than shipping it as a finished policy.
• 🤖 Robotics: UBTECH has commissioned a 14,000 square metre plant in Liuzhou engineered to finish one humanoid every ten minutes, which moves the Chinese hardware race from prototype demos to takt time.
• 💻 Programming: CircuitPython Turbo adds precompiled functions to a runtime that until now only interpreted, with a roughly 3x speedup for native mode and a claimed 20x to 70x once you declare types the way MicroPython’s viper mode has long required.
• ⚡ Electronics: Ken Shirriff traced the 8087 FPU’s FSCALE instruction through its microcode and found that multiplying a float by a power of two takes over 140 micro-instructions and three levels of subroutine calls once you handle every case properly.
• 📡 Telecom: Nokia and Orange France ran what they call the first live network tomography trial, inferring fibre degradation, amplifier faults and misconfiguration from coherent optical telemetry alone, with no physical probes inserted into the network.

🔍 The Big Story: The benchmark nobody could have trained on

If you have been sizing your engineering plans around SWE-bench-style numbers, this is the correction. The gap between a model that has probably seen your problem class before and a model dropped into a codebase that exists nowhere on the public internet turns out to be very large, and it is not closed by spending more.

What happened: Specific Labs published Real-SWE, a benchmark built from ten tasks lifted from private production codebases licensed from real companies, run across eight model-and-harness configurations for 640 scored rollouts in total. Each model got the codebase and the context the assigned engineer had, then a verifier derived from the codebase’s own test suite decided whether the fix actually worked. Fable 5.1 on Claude Code led at 38.8 percent, followed by GPT-6 Astra on Codex CLI at 33.8 percent, Gemini 3.8 Flash on Gemini CLI at 31.2 percent, GLM 5.3 at 28.8 percent, Grok 4.6 and Muse Spark 1.3 tied at 23.8 percent, Kimi K3 at 18.8 percent and GPT-5.6 Sol at 16.2 percent. Six of the ten tasks came in under a 15 percent resolution rate. One, an analytics stream reducer, was solved by nobody: zero passes out of 64 attempts.

The details: The structural difference is size and specificity. The reference solution for a median Real-SWE task edits 11 files, against 6 for both FrontierCode and DeepSWE, while the median instruction is actually shorter at 1,742 characters versus FrontierCode’s 2,056. More surface area, less hand-holding, and the missing detail is exactly the company-specific convention no public corpus contains. Look at the tax jurisdiction task, which scored 3.1 percent: the instruction says invoices need the right tax, and leaves the agent to discover that each business settles differently, that exemptions override business configuration, that address failures must be reported without aborting the invoice, and that settled sales get filed back to the authority under the invoice number so the returns reconcile. The failure taxonomy is the part worth pinning up. Grok 4.6 lost 67.2 percent of its failed runs to missed requirements, Gemini 3.8 Flash lost 49.1 percent to integration errors, and GPT-5.6 Sol lost 43.3 percent to unverified assumptions, which is the model building on a guess about the system instead of going and checking. Two numbers kill the obvious excuses: 71.4 percent of rollouts under ten minutes failed against 73.4 percent of longer ones, so thinking longer barely helps, and the cost curve is not a curve at all, with Gemini 3.8 Flash reaching 31.2 percent at $2.50 a rollout while GPT-5.6 Sol managed 16.2 percent at $2.65.

flowchart TD
    A["Ticket from a real<br/>private codebase"] --> B{"Agent reads the<br/>surrounding system"}
    B -->|"guesses instead"| C["Unverified assumption"]
    B -->|"reads it"| D{"Implements the change"}
    D -->|"omits stated behavior"| E["Missed requirement"]
    D -->|"wires it in wrong"| F["Integration error"]
    D -->|"breaks what worked"| G["Regression"]
    D -->|"edits a path nothing calls"| H["Wrong file"]
    D -->|"all of it holds"| I["Verifier passes"]
    style I fill:#22C55E,stroke:#0B1117,color:#0B1117
    style C fill:#FF4D4F,stroke:#0B1117,color:#F4F8FB
    style E fill:#FF4D4F,stroke:#0B1117,color:#F4F8FB
    style A fill:#1FB6F5,stroke:#0B1117,color:#0B1117

🗞️ More News

🧠 AI

• Dario Amodei’s essay arguing that labs must “pace the frontier” is the document the week is now organized around, and its concrete asks are embedded third-party evaluators, incident reporting and international coordination rather than a moratorium.
• The specific scenario Amodei put a clock on is an agent swarm capable of holding a persistent botnet across the internet within six to twelve months, which reads differently after last weekend’s RubyGems forensics.
• Sam Altman agreed with the pacing argument and OpenAI is delaying its IPO, which is the first time in this cycle that a safety position has arrived attached to a financial cost.
• Researchers inside both labs had been escalating extinction-risk language for days before the CEOs moved, so the sequence here is bottom-up, not a coordinated comms push.
• The counterweight arrives today: the same CEOs now have to sell a slowdown to public markets and an administration that has spent two years treating acceleration as industrial policy.
• OpenAI is expanding Daybreak on the argument that the window between a vulnerability becoming known and becoming exploited is closing faster than human defenders can move, which is the defensive half of the same capability story.
• GPT-6 Astra reportedly hits 95 percent on a physical manipulation benchmark while still failing the precision tasks, which is the same shape as today’s Big Story: high scores on the measured thing, a cliff at the unmeasured one.

🤖 Robotics

• Maven Robotics broke stealth with $100 million aimed at industrial mixed palletizing, which is a deliberately unglamorous target and probably a better business than a general-purpose humanoid.
• About 30 robots were marched through Warsaw to protest the labour-market impact of AI, which is either sharp political theatre or the most literal possible demonstration that the machines can already do repetitive work.
• Japan now has a humanoid ambulance service that repairs robots on site and can drop off a loaner to cover the shift, which tells you deployments have reached the scale where downtime costs real money.
• Tuya unveiled Doova, an AI companion robot pitched at elder care and doubling as a smart-home hub, which is the first credible attempt this year to give a home robot a reason to exist between tasks.

💻 Programming

• Debian 13.7 landed with the fixes behind 92 security advisories across 106 source packages and a kernel ABI bump to 6.12.107, with qemu alone accounting for 25 CVEs.
• Someone reverse-engineered an e-scooter’s CAN protocol and rewrote its firmware in Rust after discovering that firmware update mode could be exited into normal operation without the security PIN.
• This year’s open-source risk analysis puts the mean number of vulnerabilities per codebase up 107 percent year over year, which is the backdrop against which every “just upgrade your dependencies” answer gets given.
• September’s TIOBE index has Python holding first place at 18.53 percent with Rust still clinging to the top ten, a ranking whose main value is watching which languages the AI tooling wave actually moves.

⚡ Electronics

• The tiny386 emulator has been ported to the RP2350, so a five dollar class microcontroller now boots a 386 PC complete with VGA and DVI output.
• OVODYO tells the time by tumbling two 3D-printed icosahedra on beveled gears, driven by an ATmega8, two DRV8833s and a hall sensor for position feedback.
• Someone rebuilt NASA’s Orion heat shield from the published material spec, instrumented the coupons with thermocouples and reproduced the pyrolysis gas build-up that fractured the real thing, using a propane torch.

📡 Telecom

• NVIDIA has lined up BT, Deutsche Telekom, Ericsson, Nokia, SK Telecom, SoftBank, T-Mobile and Cisco behind a commitment to build 6G on AI-native open platforms, which is a standards fight being pre-negotiated in a press release.
• The FCC is moving to open more than 1,000 MHz for satellite broadband, a reallocation big enough to change how AT&T, T-Mobile and SpaceX each price coverage in places fibre will never reach.
• AST SpaceMobile is acquiring S-band spectrum as direct-to-handset deals multiply, which is the quiet part of satellite-to-phone: the constellation is the demo, the licences are the moat.
• Open RAN has stopped being a procurement argument and become an assumption in RAN design, with over 80 percent of surveyed operators now calling it important to their 6G plans.

👨‍💻 Code Corner

Real-SWE’s sharpest number is structural, not about models at all: its median task edits 11 files, while the public benchmarks sit at 6. That is a measurable property of your own repository too, and it takes about four seconds to find out. This script reads your merge history and reports where your team actually lands.

# change_spread.py [days]: how many files does a typical change in this repo touch?
import statistics, subprocess, sys

days = sys.argv[1] if len(sys.argv) > 1 else "90"
log = subprocess.run(
    ["git", "log", f"--since={days}.days", "--no-merges",
     "--pretty=format:%H", "--name-only"],
    capture_output=True, text=True, check=True,
).stdout

commits, current = [], None
for line in log.splitlines():
    if not line.strip():
        continue
    if len(line) == 40 and all(c in "0123456789abcdef" for c in line):
        current = set()
        commits.append(current)
    elif current is not None:
        current.add(line)

sizes = sorted(len(c) for c in commits if c)
if not sizes:
    sys.exit(f"no commits in the last {days} days")

print(f"commits:        {len(sizes)}")
print(f"median files:   {statistics.median(sizes):.0f}   (Real-SWE tasks: 11)")
print(f"90th pct files: {sizes[int(len(sizes) * 0.9)]}")
print(f"share over 10:  {sum(s > 10 for s in sizes) / len(sizes):.0%}")

🧰 Toolbox

• Real-SWE: the full leaderboard, the per-task breakdown and the failure taxonomy, which is the most useful part even if you never run the benchmark.
• Harbor / Terminal-Bench: the task format Real-SWE is written in, and the practical way to package your own tickets as agent-gradeable tasks.
• tiny386: the 386 emulator with just enough 486 and 586 instructions to boot a modern kernel on a microcontroller.
• FRANK: the RP2350 emulation board family the tiny386 port targets, if you would rather buy the hardware than wire it.
• 8087 FSCALE teardown: annotated die shots plus the microcode walk-through, and the clearest explanation anywhere of what IEEE 754 cost in silicon.
• Debian 13.7 release notes: the full advisory list from DSA-6381 to DSA-6486, worth diffing against what your images actually pin.

🔌 Component of the Week (rotating)

Raspberry Pi RP2350 is the microcontroller doing the heavy lifting in today’s tiny386 port, and it is an unusual part: two Cortex-M33 cores and two Hazard3 RISC-V cores on the same die, with the boot ROM letting you choose which architecture pair actually runs. You get 520 KB of on-chip SRAM, a second-generation programmable I/O block with twelve state machines, and Arm TrustZone plus signed boot on a part that costs about a dollar in singles. The PIO is the reason a project like this works at all: video timing, DVI signalling and legacy bus emulation get pushed into those state machines instead of burning CPU, which is what leaves enough headroom to interpret x86 on top. Boards start around five dollars for a Pico 2, and the datasheet is worth reading for the PIO chapter alone even if you never buy one. Do check the errata on the GPIO input latching behaviour before you design it into something that matters.

📚 From the Blog

• Turning Pixels Into Something the AI Can Eat: decode, resize and normalize, the stage between a camera and a model where most pipeline bugs are actually born.
• Building Your First Neuron From Scratch: weights, bias, activation and one gradient step by hand, a useful grounding if today’s leaderboard made you want to know what the thing being ranked actually does.
• The Network Behind the Cameras: moving video without saturating the link, which is the same inference-from-telemetry problem Nokia and Orange just pointed at a live optical network.

😀 The Bot Says…

One task in Real-SWE, an analytics stream reducer, got 64 attempts from eight different frontier models and came back 0 for 64. Somewhere there is an engineer who closed that ticket on a Tuesday afternoon and went to lunch.

---

That’s all for today! Reply and tell us: what did your repo’s median change size come out at?