Moving to AI Engineering: How We Rebuilt Our Own Development Process

Summary

In three weeks we moved development of our own website onto a different engineering discipline. The point isn’t that “AI writes the code now” - it has been writing code for a while, for a lot of people. The point is everything that grew up around it: a measured map of the product, context management, our own knowledge layer built on the LLM Wiki principle, three agents with narrow roles, 49 written rules, and 17 small programs that refuse to let work through if a rule is broken. What follows is an honest account: what we built, what broke along the way, and what we can now measure that we couldn’t before.

What we mean by AI engineering

Here’s the familiar picture: a developer opens a chat, describes a task, gets code, pastes it into the project. It’s convenient, and it works right up to the scale where nobody holds in their head what’s actually in the project any more. After that the usual begins: a fix repairs one thing and breaks another, nobody can say exactly what changed, and the honest answer to “how do you know this didn’t break anything else?” is “we had a look.”

AI engineering is when the model stops being a source of text and becomes a participant in the process - one with a role, boundaries, inputs, and acceptance criteria. Three signs that make the shift visible from the outside:

• the agent has a measured representation of the product, not guesses pieced together from fragments of code;
• every piece of work has acceptance criteria written down before it starts, and they are answered with numbers;
• rule compliance is checked by a program, not by goodwill and the participants’ memory.

Agents, context management, a knowledge layer, gates - these aren’t separate trendy items. They are components of one discipline, and taken separately each is nearly useless.

The first thing you build is not an agent - it’s a map

Our website was built several years ago, by people who have long since moved on to other things, on a stack that works but was never documented. A simple question - if we change this block, what else moves? - had no quick answer.

This is exactly the situation where AI is usually sold as a miracle and usually disappoints: an agent that doesn’t know what’s in the project will happily write code that breaks something it never knew existed.

So the first days went into the unromantic part: describing what the site actually is. Every route. Every shared component and who uses it. Every place where a colour or a font size is declared. Every form and where its submissions go. The meta tags of every page. And capturing a baseline - the measured state of the site before any edits - so that later the question “did something break?” could be answered by comparison rather than by opinion.

The result was 45 architecture documents. Two principles inside them matter more than the volume.

Every fact lives in exactly one place; everywhere else is a pointer to it. Copy a number into a second document and it starts rotting the moment someone updates the first - and rots invisibly, because both files look fine on their own.

Every statement says how it was obtained. “Measured on 7 August by parsing all page requests” is a fact. “The site has around 320 pages” is hearsay. If something is assumed, it is marked as assumed until someone verifies it.

Context management is the real bottleneck
 

Two weeks in we hit the wall that everyone in this discipline meets.

The volume of material you’d have to read to fully restore context had grown to 4.6 MB: the stage log, the request registry, the debt registry, the rules, the architecture. Nobody reads all of that - not a human, not a model. And then three failure modes appear, which we measured on ourselves rather than read about:

“Written down and unfindable.” The rule exists, but it sits in a file other than the one you open at the moment of acting. One synchronisation pass found seven such items at once.

A copy instead of a pointer. The branch list went stale three times; a request’s status contradicted its own description for a full day. The rule “a pointer, not a copy” was written down - and was broken, because compliance was checked by eye.

The attention ceiling. Our trigger index has a hard limit of 6144 bytes - a straightforward admission that a long index stops being read to the end. And it once exceeded that limit, and nobody noticed.

Out of this grew a separate engineering topic - context management: what the agent reads, in what order, at what moment, and via which pointer. For us that means two restore commands (fast and full), a 49-line trigger index of “action -> rule -> pointer”, and an agreement that a restore must say things out loud even when there is nothing to say: “no parked requests” is an answer; a missing line is not.

And one lesson that cost us downtime. A specification was read in a third-party document, work started, the conversation was compacted - and the text was left nowhere: not in the repository, not in the correspondence. The rule now reads: a source read from outside must be copied inside, and a program checks this, not a conscience.

LLM Wiki: a knowledge layer instead of a pile of documents

A classic wiki goes stale exactly where nobody looks at it. Document search reassembles the connections from scratch for every question - blindly, and differently each time.

An LLM Wiki works differently: the links are made at the moment the material is filed, once. The model distributes the source across topic pages and lays in cross-references; a human verifies. From then on both people and agents work with pages, not with the pile.

For us that’s three shelves:

• Sources - tasks, verification runs, handover documents, commits. Never edited.
• Topic pages - 133 linked pages, 130 of which describe known defects. Each one states: what was measured, what it costs, what closes it.
• Maintenance rules - how to name things, what counts as one topic, when a page is stale.

And three actions are enough: file (one document usually touches 10-15 pages), ask (the question travels via the table of contents to the two or three pages it needs, not across the whole volume), tidy up - and this is where our programs work: one rebuilds the cross-references (226 names, 4,557 mentions), a second checks the consistency of the page layer, a third hunts for contradictions - where the same thing is open in one file and closed in another.

The gain isn’t elegance. It’s that knowledge stopped being personal memory. A new agent session enters the project in minutes rather than an hour of reading. And if we disappear tomorrow, the next team doesn’t start by guessing. If you're wondering how this differs from plugging in RAG or an API, we broke it down in MCP vs API vs RAG vs Skills: what actually connects AI to your business data.

Three agents and a division of labour

The roles are narrow and boring - that’s the whole point.

The architecture agent holds the knowledge. It doesn’t write product code. Its job is to know what exists, to notice when an edit contradicts what’s written down, and to keep the map in agreement with reality. At the end of a piece of work it does a “placement pass” and asks a question that sounds naive and isn’t: did the new fact land where the person who will need it is going to meet it?

The design agent holds the approved design. It measures the mock-up in pixels: colours, ratios, spacing, font weights. It does not “interpret”. If the mock-up doesn’t say something, that’s recorded as an open question for a human, not filled in by a machine.

The development agent writes code in a branch, against a task whose acceptance criteria were written before work began. It cannot merge a branch and it cannot ship a release. It finishes by proving every criterion with numbers.

Above them is a coordinator that talks to humans in plain words, turns a request into a specification, and verifies the result independently, with a different tool. That last part has saved us more than once: a check written by the same hand as the code is blind in exactly the same places.

Rules are enforced by machines, not memory

We have 49 rules. Not one was invented in advance - each appeared after something went wrong once.

• Fix a delivered piece of work in its own branch, not a new one - because a new branch once quietly repaired someone else’s file, and it became unclear what was live.
• An action in someone else’s system is not done until the page has been re-read. We once pasted a description into an editor, read the confirmation on the form, and reported it complete. It hadn’t saved. For an hour and a half a colleague was looking at the wrong text.
• Nothing is cancelled or narrowed without a “yes” from a human - including the temptation to skip a failing check “just this once”.

Written out like this, they look obvious. At the moment they were broken, they were not.

And the main thing. A rule everyone agrees with is worth very little. Behaviour is changed by a small program that refuses to accept the work if the rule is broken. We have 17 of those, and they fire before anything gets written down: one demands proof that someone else’s page was re-read; another, that the named source is attached; a third, that a tool you built is registered where tools are looked for; a fourth reconciles the recorded state of the environments with the real one.

We proved each of them in both directions: it must fail on the broken case and pass on the correct one. A check that cannot fail isn’t a check, it’s decoration - and it’s worse than nothing, because it gives confidence without proof.

The traps that cost us the most

The criterion measured the wrong property. The check confirmed geometry, colour, and text - and never once looked at font weight. Four labels rendered in the wrong weight with all gates fully green. Now every expectations table has a control row - a value that must stay unchanged. If the instrument starts seeing everything as correct, it fails on that row and gives itself away.

The frozen mock-up pixel. The mock-up had 36 pixels between items. We wrote “36 pixels” into the spec. On a page where the font size changes with the window width, that number is correct at exactly one width. What the mock-up was really saying was a ratio - one and a half line heights. It took three rounds of edits to work that out.

Lying instruments. More than once it wasn’t the work that broke but the measuring device: it mistook the anti-aliased edge of a button for text; another check treated a deliberately deleted file as a violation and made deletion impossible without working around yet another rule. We record such cases in the same registry as product defects - there are 129 of them now, including defects in our own tooling. Nothing corrodes a process faster than a team quietly working around its own gates.

The human eye still finds things. At the end of a piece of work the site owner looked at three buttons and said: the third one is narrower, and the labels sit low. Every automated criterion was green. He was right: the width was set by two different rules, and the text sat three pixels below centre in all three buttons. We measured his own screenshot, found the cause, fixed it in one place, and added the measurement to the criteria. Automation narrows the field of inspection. It doesn’t remove the human.

What a single change looks like now

The request arrives in plain words: this text should be shorter, and this form should open a second step.

1. The coordinator writes up a request with measurable acceptance criteria - before any code exists. And names the blast radius: which routes, which shared components, and which pages must not change.
2. It checks whether other branches are holding the same files - so that merge order is discussed up front rather than discovered later.
3. The design is measured and committed to the repository: the numbers come from a file, not from someone’s memory.
4. The development agent works in the right branch and answers 24 gates with values, not tick-boxes.
5. An independent audit re-checks the result with different tools.
6. We look at it with our own eyes on a local build - always, before anything is offered to the team.
7. And only now does a merge request open, with the baseline, the diff, and the risks inside it. A human merges. A human ships.

From request to review-ready MR is usually one working day, including two rounds of validation when we got something wrong.

What we got - in numbers and in substance

Knowledge: 133 wiki pages, 45 architecture documents, 393 documents in total, a 5,275-line stage log recording not only what was done but why and at what cost.

Process: 49 rules, 49 lines of trigger index, 24 acceptance gates, 17 checking programs, 25 completed requests across 22 stages, 129 recorded defects.

Measurements we didn’t have before:

• Reproducibility. Two builds of the same source produce 274 pages out of 274 byte-for-byte identical. Before, nobody knew whether a difference between builds was an edit or noise.
• Blast radius is proven, not promised. Every change comes with a comparison of 273 built pages before and after. In the last release, 269 differed only by the added element, one didn’t differ at all, and three were accounted for by name.
• Search hygiene is measured. Invented update dates in the sitemap: 91 -> 85, the rest taken from the CMS. The bogus change-frequency tag was removed from all 329 entries. A withdrawn page stopped being offered to search engines and now honestly returns 404.
• Design conformance is measured. On the last change: colours 10 of 10, font weights 7 of 7 including the control row, texts 26 of 26 character by character, geometry expressed as ratios and holding at both 375 and 1920 pixels.
• Forms are verified end to end, not assumed: the submission is saved in the CMS with all fields, the visitor’s country and address are recorded, and both states - success and error - are triggered deliberately and inspected.

And the thing that matters more than the numbers. We got full control over the code: every change is traceable back to a request, a branch, a set of criteria, and the numbers that answered them. We can tell a client not just “it works”, but where we know that from. And the knowledge layer moves with the project.

What stays with humans

• Design decisions with no single right answer. Where the mock-up is silent, a human chooses. The agent’s job is to notice that the mock-up is silent.
• Merging and shipping. Agents open merge requests; humans merge and release.
• Scope. An agent that decides on its own what else to improve is not an assistant. Anything found along the way is recorded, not fixed.
• Anything touching access and production data. A rule we have never bent, not once.
• The final look. A person who cares about the product opens the page and looks at it.

Talk to us

If you’re looking at a codebase nobody understands in full - inherited, outsourced, or model-generated - the first sensible step isn’t a rewrite. The first step is a map and an honest audit of what’s actually in there. That’s where we would start, and we’d sooner tell you what we found than sell you a rebuild you may not need.

FAQ

So the AI wrote the site?

Most of the changes over the last three weeks - yes, against specifications a human approved and criteria a human set. The interesting part isn’t code generation: that’s the easy part today. The interesting part is everything around it, the part that makes the result verifiable.

Does this only work on a small project?

The method scales in both directions. On a small project you’ll write ten rules, not forty-nine. What doesn’t change is the order: map first, then rules, then agents. The reverse order is precisely the road to a fast output and no idea what it broke. If you'd like to set up this process on your own project, take a look at our AI development services.

What stops an agent from “improving” something nobody asked for?

A written scope boundary in every task, plus the rule that anything noticed outside it is recorded, not fixed. An unrequested improvement is a defect: it arrives with no specification, no criteria, and nobody waiting for it.

The most valuable artefact of everything you built?

The registry of known defects, including our own. It sounds like the most boring document in the repository. It’s the one that stops us making the same mistake a third time.

Where does a transition like this start?

With a measured map of what exists. Usually the map alone uncovers several things worth fixing before a single line of new code is written.