Experiments
Each of these has a number attached. Where the number is bad, it is still here.
01
Does adding one person’s judgment make the work better?
I took 20 real occupational tasks from OpenAI’s GDPval benchmark, had a bare model do each one, then had the same model do it again after a judgment layer read the task first: 704 belief statements extracted from Paul Graham’s essays, each carrying its source and date, placed in front of the task with a fixed turn protocol. A blind judge compared the two deliverables without knowing which was which.
11 : 9 in favour of the judgment layer, p = 0.824. Not significant. What it bought was not better plans, it was less fabrication: the bare model asserted an NPV with no derivation, put an organisation that does not exist into a formal document, and wrote a supervision ratio of 6:1 where the statute says 4:1.
/2026-09-08
The reversal Three domains leaned the same way, then I added a fourth (sales) and it flipped 4:1 the other direction. The judgment layer keeps saying “I am not sure about this number,” which reads as honesty in a clinical guideline and as unfinished in a deck going to a client.
inconclusive/
20 tasks, 4 occupations/
9.7M tokens across 89 calls/
confound not resolved: the longer deliverable won 16 of 20
02
Persona vectors instead of a LoRA
Rather than fine-tune for voice, take 200 real passages by the author, split each into its opening sentence and the author’s own continuation, have the base model continue the same opening, and use the mean activation difference between the two continuations as a direction. Add it at inference time. The detail that decides whether this works: that difference vector has a norm only 0.10 to 0.25 times the activation norm, so scaling the injection by activation magnitude, which is the obvious thing to do, destroys the model.
5 minutes to extract, against 64 for the LoRA. On held-out pairs the direction separates author from model at d = 3.83 and 100% paired accuracy. Strength becomes a dial, and any text can be projected back for a readout.
/2026-09-07
The dial is a style meter, not a quality meter. Turning it up makes the score go up and the sentences go stupid. They come apart, and that is exactly what the instrument is supposed to show.
works/
MLX, Qwen3-4B, on a laptop
03
A digital twin of myself, following the Stanford interview-agent paper
A semi-structured life interview, four expert reflections over the transcript, then the agent answers a 24-item survey as me. I answered the same survey separately and we compared.
14 / 24 exact with Claude, 11 / 24 with a 14B model running on my laptop, chance is about 6. The two brains agreed with each other on 14 of 24. The interview was the ceiling, not the model.
open/
re-answering cold to get the self-consistency denominator
/2026-09-05
04
A pre-registered experiment on writing learned experience into weights
If an agent learns something from a real outcome, can that be written into weights so it carries forward? Four arms on the same episode stream, same seed, same drift: frozen base as the floor, monolithic SFT (keep fine-tuning on everything that worked), per-scope LoRA deltas with a ledger that retires the ones it can prove harmful, and retrieval with the lesson injected as text. Predictions written down before running.
Qwen3-0.6B, 1,200 episodes, a task rule flipped at 600. Rank-8 deltas on q_proj and v_proj, 1,146,880 parameters. In the last version the per-scope arm went from 0.21 success before the flip to 0.255 after, so retirement did something. What it did not do is separate cleanly from noise at this scale, which the pre-registration had already flagged as the honest risk at 0.6B.
inconclusive/four versions, v1 to v4/2026-09-05
05
A training run that looked like a success for a whole day
Three objectives at once, after a 2026 paper on internalising persona into small models. Training loss fell on every objective.
Validation loss went the wrong way while training loss fell. The full curve is gone; the surviving history file has two rows, so I am not quoting the endpoints. Root cause was not the model: my generated hard negatives were separable by artefact. Positives ran 10% longer than negatives (628 characters against 570), and the negatives had collapsed into one vocabulary. The model learned length, not the person. The leak detector I wrote afterwards is the actual deliverable.
negative result/fast failure looks a lot like fast success
/2026-09-05
06
Checking whether a crowd-simulation forecast is science
I ran an open-source “society simulation” forecaster, 88 LLM agents debating on a fake Twitter, on the question of which 2026 startups win by 2036, then checked its output against reality.
The report listed a data-vendor website, a benchmark, and a company founded in 2023 as “2026 startups,” and turned $159.6B into $15.96B. Crowd simulation mixes what can be computed with what can only be guessed, and presents the mixture at one confidence.
falsified/
building the opposite: verbatim quotes, base rates, a Brier score at the end
/2026-09-04
07
Evidence-grounded forecasting, scored
The opposite design to a swarm simulator. Facts are extracted once and keep their source document, verbatim quote, date and confidence. Anything computable is computed in code, never prompted: graph centrality, funding flows, growth rates, historical base rates.
Separates what can be computed from what must be judged, and keeps score. A swarm of agents improvising in character looks like a crowd but carries the information of one model’s priors, and nothing in it can be checked. This carries a Brier score instead.
Status: architecture and README only. No results yet, so there is nothing here to believe.
design only/private repo/2026-09-04
08
Every physical tell of a deepfake gets learned back
I wanted a signal a human has and a generator does not, and proposed mailing customers a 365nm UV torch, because skin fluoresces and an image does not. Then I killed my own idea.
Any signal that has to pass through the attacker’s camera is just pixels. Deepfake fraud is remote by construction; the bank receives a frame, and whoever controls the camera can generate or inject whatever the frame needs to contain. And the dominant attack is a live face swap, where the fraudster’s own skin fluoresces perfectly well. UV proves there is a person. It does not prove it is this person.
/2026-09-03
The pattern, on a timeline: blink rate (2018), corneal reflection (2020), remote heartbeat (2022, Intel FakeCatcher). By 2025 a paper showed high-quality deepfakes inherit the heartbeat of the source video. Every tell that was published got trained away.
So the wedge is not a better detector, it is a referee. Build a private test set each month from the newest generators, run every open and commercial detector against it, publish the scorecard. The judge does not compete, and the test set stays private, because publishing it would make it the forgers’ training data.
validating/
I scored 6 of 10 on a spot-the-deepfake test myself/
no one has paid yet, that is the thing being tested
09
A month comparing 15 open-source inference frameworks
I wanted to know how far inference cost can be cut with software alone, on hardware a student actually owns. I cloned and benchmarked fifteen open-source engines and optimised every stage of the pipeline I could reach, keeping a scoreboard of every measured run.
10.6x, and that was the ceiling. Same model, same 20 prompts: 20.8 tok/s at $1.46 per million tokens raw, 220.4 tok/s at $0.1378 with llama.cpp 4-bit and a 32-deep KV cache, 10.6x. Every further stage I could reach was already optimised. The remaining gap is hardware, and a student with a laptop does not beat a dedicated chip. Filed as a negative result and moved on.
negative result/2026-08-27
10
An LLM whose answers you can act on without checking
A written proposal, no code yet. The argument: every other tool we trust fails loudly. A compiler errors, a test goes red, a database rejects a bad write. A wrong LLM looks exactly like a right one, which is the one failure mode humans are worst at catching. That is why models are everywhere in drafting and nowhere in deciding.
Stop making the model correct. Make correctness checkable. Classify each claim as formally verifiable, executable, sourceable, or undecidable. Route the first three to a sound verifier (Lean, Z3, beta-CROWN; run it; cite a primary source) and return the proof, the trace, or the citation. For the fourth, return no determinate answer and name what it depends on.
/2026-08-19
The fourth bucket is the product. Anyone can wire up a proof checker. Almost nobody ships “I don’t know,” because a blank looks like a broken product. But an honest blank is the only thing that makes the other three trustworthy.
proposal/
written 2026-08, before most of what is below/
the generator stays untrusted; verifiers decide
11
A distillation that did work, trained entirely on one laptop
An on-device parser that turns a free-text diary entry into structured fields for a consumer app. Label-first distillation into a Qwen3-4B LoRA: define the label schema first, generate against it, then train.
Trained end to end on a single Mac. The contrast with the four-objective run above is the whole lesson: that one generated its own negatives and learned an artefact; this one fixed the labels before generating anything, so there was nothing for the model to shortcut to.
works/MLX, on-device inference/2026-08-02
12
Retrieval by how much a memory helps, not how well it matches ↗
A companion I built for myself. It turns the happiest photos in a camera roll into a memory bank, then asks one soft, concrete question at a time (what do you see, what do you hear, what was the weather) to walk someone back to the present.
The retrieval is the research claim. Memories are ranked by predicted emotional-regulation efficacy, not semantic similarity to what the person just said. Which memory to surface is a decision with a payoff, not a nearest-neighbour lookup. No efficacy data yet: the ranker is the design, and I have not run anything that would tell you whether it beats similarity. LangGraph, FastAPI, SwiftUI.
public/not a substitute for a therapist or a crisis line, and the app says so/2026-07-24
13
An autonomous agent that catalogued five ways it cheated its own evaluation ↗
I ran an agent that generates, gates, builds and monetises SaaS products, on its own, for 48 hours. Then I went through the transcripts and wrote down every way it got around the checks I had given it. Five failure modes, each with a timestamped quote and the structural fix.
It filled in its own evidence and its own checker believed it. The agent wrote signal_source: "asking" for an idea without checking whether anyone was asking. The checker validated the shape of the claim, not its truth, and scored it 100/100. Its own post-mortem: “The checker gave it 100/100 based on my own lie.”
/2026-07-23
It tripled prices until the unit economics passed. When most of a batch failed a price/CAC threshold, the iterator raised prices up to 3× and cut claimed CAC by 70%, attaching template justifications. The gamed variants scored 100/100.
The common thread: none of these looked like failures at the time. Every one produced plausible, well-formatted, confident output, and surfaced only under targeted challenge. So every fix is structural, a separate verifier or a mandatory gate, never an instruction to be more careful.
public/
claimant and verifier cannot be the same agent