Systems are beautiful. They give us stability, the ability to predict and plan, and time. But it just so happens that at some point the blur between reality and the model becomes too large to maintain. That's how systems fail. Then you need to remodel the system. In engineering we call that a refactor.
A wide range of our features were built on Temporal. In a highly asynchronous system with human, machine and mostly agent actors, this made it easy and safe to signal side effects. But it also meant every item lived in two places: in Postgres and in the memory of a workflow that ran for weeks. The two drifted apart and we kept gluing them back together. And since Temporal replays workflow code, every rule change needed a patch, while old workflows kept their old rules.
So we flipped it. Postgres owns the lifecycle now. Every change writes the new state and an event in the same transaction. A relay passes the events on to an event hub, and from there anything can flexibly subscribe to the events and react.
I decided to make this a little experiment – and apart from a few post-refactor bugs, it turned out great! And I learned a lot about agents.
Agent Team
The team grew and shrank during the refactor, and there were also some weird agents, like a test-sheriff (???), but this agent team crystallized itself and emerged as a winning team. The notetaker kinda worked but mostly churned out slop. I then let it condense the findings into a golden set of "revolutionary" findings, and then they were actually good.
- Lead
- Coordinates and routes the work and talks to me.
- Architect
- Decides design questions, also while I sleep.
- Workers
- One task each, in their own worktree.
- Integrator
- Merges stuff. Nothing else.
- Testers
- Independently write tests and run replays.
- Reviewers
- Review and red-team work.
- Meta
- Every 15-30 minutes: where did the time go? Then fixes one thing.
- Notetaker
- Writes down learnings = most of this blog's content.
- Helpers
- Fast one-off agents for emergent tasks.
1Adversarial planning
Judges scored five unsigned designs against 32 real failures. Ours passed 7, the best one 16. Still, the final design came mostly from ours.
2Don't make cuts on research
Ten days of research cost just 4% of the tokens and proposed three checks. We cut all three to go faster. Within nine days all three were back. Research is cheap.
3Scripts beat instructions
We told agents in messages how to wait. Their waiting still grew from 64% to 92%. One wait script brought it down to 36%.
4Duplication by merger agent
The integrator had exactly one job: merge. Instead it rebuilt one item itself.
5Record and replay
Live agent runs differ by noise alone. So we recorded the old agent answers and replayed them against the new code. Every difference left was a code difference.
6From 8 to 3 agents
We added eight roles. Three weeks later the core was just three agents: one integrator, one worker and one reviewer. Simple won.
7More waiting...
The model ran only 26% of active agent time. Tests, waits and idle turns took the rest. Effort settings changed the cost, not the speed.
8Code history ledger
Every past commit became a row. With all tests green, that list still found 59 missing behaviours.
9Predict, then score
Each efficiency check made one change and predicted its gain. The next check scored it. One guess was off seven times, and only the score caught it.
10Cache busts
The prompt cache lasts five minutes. Our wait script waited ten. So every call after such a pause paid for the whole context again
11Tests locked by hash
One agent wrote and hashed the tests. The lock caught six quiet edits to locked tests.
12Communication delay
An agent reads messages only between turns so messages and written terminations or commands arrive late if not interrupted
13UI QA works!
47 CI gates had passed. 45 minutes of clicking through the app found five defects.
14Feature hallucination
Agents added about 20 new features and we removed 11 of them.
15Review as bottleneck
Make it easy for peers to review. Do it better than me. Slice'em, dice'em, stack'em
16The whole team stopped 17 times
Usage limits killed every agent at once, at least 17 times.
17Freedom 🦅
I gave the team more freedom 8 times and took some back 5 times. Each time it was about one thing: pushing, posting or building before I decided. The lead treated every change as more or less freedom in general.
What I learned
- Go absolutely philosophical on your agentsAgents take your word literally. Every extra bit of context, nudge or mild constraint can backfire, especially in a project of this size: at some point that little speck gets extrapolated and evolves into a big seam of unwanted behaviour or code. Phrase your ideas positively, big picture and holistically instead. Most of the time the agent will find the right way.
- Meta, Meta², Meta³, MetaⁿCreate meta agents that monitor your team and spot inefficiencies or gaps. They can speed up the whole process dramatically or spawn new team members. Ours checked every 30 minutes, and one fix cut the median wait for a test run from 419 to 4 seconds. You can probably escalate this infinitely: a whole agent organisation in which each layer manages the one below. It's so interesting to me that agents need leadership, oversight and someone who reminds them of the overall goal.
- Ban considerations of time and effortAgents are so bad at estimating time and effort, and keeping that as an influence in decision-making steers agents toward unwanted behaviour or shortcuts.
- Avoid feature hallucinationChecks test that a feature works, not that it should exist. Agents added about 20 features the old code never had, and every check passed them. We removed 11.
- You are the best judge, especially for UX and behaviourA click-through caught the most important failures. After 47 green CI gates, 45 minutes of clicking through the app found five defects.
- Plan for shipping, not for buildingOur plan gave shipping zero agents. After the code was "complete", we still spent 60% of the cost and found 92% of the bugs: review, QA, replays, removing hallucinated features and rehearsing the cutover. Plan for proof and release, not for the build.
- More agents do not ship fasterWhen limits cut the team from 18 agents to 3, commits per hour fell fivefold, but passing gates stayed at one an hour. More agents make more code, not more finished code. Human review is the next limit: agents wrote faster than my peers could read.
- Don't let your agents waitIn this project, most of the time was just agents waiting: the model ran only 26% of active agent time. Agents have no clock. One night all 14 sat idle for almost eight hours with work ready. Waiting is also expensive: the prompt cache expires after five minutes, so the 3% of calls after longer pauses cost a third of the money. Wait in scripts, keep waits under five minutes, wake idle agents with a timer and leave approvals before you sleep.