OpenAI’s legacy-code scores jumped sharply—does that make migration safe?
OpenAI reports large gains on a 20-language SQLite task and a hidden-test COBOL migration evaluation. The test suite is public, but the results remain company-run and do not establish that production modernisation is safe.
The short version
- What happened
- OpenAI reports that GPT-6 Astra averaged an 87.4% test pass rate across 20 languages on MultiLangBench, compared with 26.8% for GPT-5.6 Sol.
- Who it affects
- Legacy-system teams have a stronger reason to run bounded pilots, but not to remove expert review or operational safeguards.
- What remains unproven
- OpenAI designed the evaluations, ran them and sells the evaluated models; there is no independent replication.
Keep in mind
OpenAI designed the evaluations, ran them and sells the evaluated models; there is no independent replication.
The article omits the COBOL migration denominator, per-target-language scores, repeated-run variation and uncertainty intervals.
The repository documents 24,000 converted SQLite tests and a recommended 272-test subset, but the article does not identify which produced the headline results.
Neither evaluation measures production defects, security, operational incidents, maintenance burden or worker outcomes.
Source analysis · 2 linked sources. Primary analysed source date: 19 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
A much stronger result on two bounded coding tasks
OpenAI reports a large jump in coding performance for languages with less public code and documentation than Python or JavaScript. MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in 20 languages. GPT-6 Astra averaged an 87.4% test pass rate, versus 26.8% for GPT-5.6 Sol. The company reports gains from 10.9% to 72.1% in COBOL, 21.4% to 83.5% in Fortran and 27.5% to 89.4% in Visual Basic.
The released repository makes the task inspectable. It converts 24,000 tests from SQLite's TCL suite into database inputs, queries and exact expected outputs. Because running all tests is expensive, the documentation recommends a 272-test subset for routine evaluations. OpenAI's article does not say whether its headline scores used that subset or all converted tests, how many complete model attempts were run, or whether uncertainty was calculated.[1][2]
COBOL migration is tested separately
A second evaluation asks models to translate COBOL programs into C++, Java, Kotlin, Python or Rust while preserving behaviour. Hidden tests require byte-for-byte parity on observable outputs; a migration passes only if every test passes. OpenAI reports 86.6% accuracy for GPT-6.1 Sol at low reasoning effort, against 5.4% for GPT-5.6 Sol, and 97.9% at high effort. It does not disclose the number or provenance of migration programs, per-target-language results, repeated-run variation or the hidden test suite, so outsiders cannot yet reproduce that headline result.[1]
The distinction that matters
What the two evaluations actually test
The company separates building a database engine from translating an existing COBOL program.
Build
Implement SQLite behaviour in 20 languages
Exact database tests measure partial functionality
Translate
Move COBOL into five target languages
All hidden output tests must pass
Deploy
Modernise a live production system
Not measured by either evaluation
Useful evidence, not permission to automate a cutover
For engineers maintaining banks, government services, weather systems or industrial software, the results support testing newer models on neglected languages. They do not show that an agent can understand undocumented business rules, live dependencies, security constraints or operational failure modes. The benchmark developer also sells the evaluated models, and no peer review or independent comparison is supplied.
A safe migration still needs a complete inventory, preserved reference behaviour, domain-expert review, representative tests, staged deployment and rollback. Evidence that would change this assessment includes independent reruns, a published COBOL denominator and per-language results, contamination checks, repeated trials with uncertainty, and prospective studies of real migrations measuring defects, review effort, incidents and total cost.[1][2]