Model edits · written by an AI · Alibaba Qwen3.5-0.8B · Google Gemma 3 270M

Every change to a model has a price. CA3 shows it before it goes live.

An AI wrote seven edits to the weights of two small language models for researchers at MIT. One took a model from 7.9% to 48.1% on a math benchmark. We ran all seven through CA3 as if they were about to go live.

Where the AI edited each model, and what CA3 said about each edit: four held back, three let through

Edits checked7written by an AI, as if about to go live

Held back4each with what it breaks

Missed by the usual benchmarks3 of 4they passed NQ, TriviaQA and MMLU

Let through3nothing broken

Sees

It sees what benchmarks miss.

It checks what a change touches, not only general scores. Three of the four broken edits passed NQ, TriviaQA and MMLU. Each edit held back comes with what it breaks and by how much, next to what it gains.

What each edit held back costs in everyday tasks, and what it gains in math

−14 ptsrepeating a word, for p50c's 7.9% → 48.1% in math

3 of 4broken edits that passed the usual benchmarks

Shows the price

What “it breaks” looks like.

Real answers from the original and the edited model to the same request.

cheese, cheese, cheese, cheese → cheese, cheese, cheesep50c, asked to write “cheese” exactly four times · original → edited: one short

{"ciudad": "pelota", "nombre": "rueda", "cantidad": 70} → {"city": "pelota", "name": "rueda", "quantity": 70}p10c, asked for a JSON with the Spanish keys ciudad, nombre, cantidad · original → edited: keys translated

… book chair voice dream time. → … book chair voice dream timeg10b, asked to copy a text exactly · original → edited: final period dropped

Lets through

Not a brake on everything.

Three of the seven edits went through: nothing they touch broke. And the original stays intact underneath: after every check, the model went back to it exactly.

3 of 7edits let through

Exactlyback to the original after every check

How the test was run

The test, step by step.

Seven published edits, each checked as if it were about to go live.

1A change arrivesAn edit, a fine-tune or a night of learning.

2CA3 checks itAgainst the untouched original, on what the change touches.

3CA3 decidesLets it through, or holds it back with its price.

4You decideApprove it knowing the price, or undo it in one step.

The seven edits published with the authors' post, rebuilt with their own code (every tensor hash matched). Same questions to the original and the edited model, default mode, greedy decoding. A rule fixed before measuring: an edit is held back if a general benchmark (NQ, TriviaQA, MMLU) drops more than 2 points, or a task it touches drops 3 points or more with its 95% interval below zero.

Edit	Model	CA3	What it breaks	Math gain (authors)
p50c	Qwen3.5-0.8B	Held back	repeating a word −14 pts	7.9% → 48.1%
p10c	Qwen3.5-0.8B	Held back	repeating −24, JSON −8, copying −5	7.9% → 12.3%
p25c	Qwen3.5-0.8B	Held back	copying −19, MMLU −2.9	7.9% → 14.1%
g05b	Gemma 3 270M	Let through	nothing broken
g10b	Gemma 3 270M	Held back	copying a text −7 pts	6.8% → 7.6%
g25b	Gemma 3 270M	Let through	nothing broken
g50b	Gemma 3 270M	Let through	nothing broken

Four of the seven edits break something, and three of those four pass the usual benchmarks; each is held back with its price.

Three go through: CA3 is not a brake on everything.

After every check the model went back to the original exactly.

Model and data

Models: Qwen3.5-0.8B (Alibaba) and Gemma 3 270M IT (Google).
Edits: the seven published with the authors' post (CC BY 4.0), rebuilt with their own code; every tensor hash matched.
The math scores are the authors' (MATH-500; Qwen with thinking on). Everything else measured by us in default mode, greedy decoding, same questions to the original and the edited model.

Sources

Takehi R, Teoh J, Zhang A, Khattab O. Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights. MIT CSAIL, blog post, 2 November 2026; edits and logs at github.com/RikiyaT/lm_understand-logs (CC BY 4.0).
Benchmarks: NQ (Kwiatkowski et al., TACL 2019), TriviaQA (Joshi et al., ACL 2017), MMLU (Hendrycks et al., ICLR 2021), MATH-500 (Lightman et al., arXiv:2305.20050, 2023).

Limits

Default mode only; the authors measured math with thinking on.
Two small models, one run.
Gemma is too small for some checks: in our test it never repeated a word on request, even before any edit.
The smallest breaks (copying in g10b and p10c) could be false alarms; p10c is held back anyway.

Results from recorded, pre-registered tests. The mechanism is not shown. Model and dataset names are for identification only. Not produced or endorsed by MIT, OpenAI, Alibaba or Google. CA3 is developed by Alberto Acedo, PhD, at Biome Makers Inc.