Home / Writing and talks / LLM lessons

What building with LLMs in production taught me

Eleven lessons from fifteen months building Work OS, an AI product that enterprises used to make decisions about their workforce. When the output goes in front of a board, "good enough" needs a precise meaning.

1. Narrow prompts beat broad ones

Our early prompts tried to return everything a user might need in one call. Coherence dropped, latency rose, and the output was harder to use. Splitting each request into focused, single-purpose prompts fixed all three. Treat a model call like a well-scoped function: one kind of input, one kind of output.

Related points. Giving the model the outcome, the role and the task was enough context; scripting every step tended to fight how it reasons. And what goes into the context is a product decision, because richer context costs both time and money. And we never needed fine-tuning: our problem was knowledge and structure, not behaviour, so grounding prompts in our own data was the right tool.

2. Latency is a design problem

Return the important answer first and fetch the extras afterwards. We moved "next step suggestions" out of the first response and into a follow-up the user asks for. Generating recommendations on first click was also too slow at the moment of decision, so common paths moved to background computation. Generate on demand only when the context has really changed, and say so while the user waits.

3. Quality thresholds are never finished

A 50% match threshold left too many screens empty. A lower one at launch brought in more matches, and the team later raised it again as the library grew. Tune against real data, and watch empty screens and user trust together.

Scores also need checking against real customer data before an executive sees them. Ours looked very different on demo data.

4. A simpler data model helps the model

Each layer of nesting in the data adds prompt complexity. I argued for a flatter model, was overruled, and was wrong about the value: the extra level turned out to be the one customers most wanted to control. The cost in prompt complexity was still real. The lesson is to manage that cost, not to assume the simpler model wins.

5. Human review is part of the product

Sign-off is a first-class flow, not an edge case. AI took over one step at a time, a person published every change, and versions only moved forward so anyone could see what changed and who approved it. Enterprises don't flip a switch. They validate one workflow at a time.

6. Measure value honestly

Time saved per run is the flattering number. Multiply it by how often the work runs, how much of the team it covers and how many people choose the AI version, and you get a number you can defend. Launch with clear estimates, instrument for actuals, and replace one with the other over time.

The loop of capture, redesign, deploy and measure is itself a product feature. Without it, AI impact stays as anecdotes.

7. Governance is what keeps the product in place

Enterprises need one view of every AI agent they run. Explanations belong at the point of decision, on the screen where someone chooses, not in a help page. And a generic agent that claims to do everything will distort any ranking unless the ranking accounts for how specific an agent is.

8. Every rejection is a signal

Capture what users accept, reject, edit and ignore from day one. It is the cheapest source of improvement you will have. Whether to show confidence scores depends on who is looking: decision-makers want them, and people doing the work often find them confusing.

9. The model writes the inputs; code writes the numbers

Every figure a customer saw, from capacity to savings, was calculated in plain code over data their own team had reviewed. The model helped produce what went in, never the number that came out. When an executive asks where a number came from, "the AI said so" is not an answer you want to give.

10. Ask for judgements you can check, not a verdict

A single score from a model is hard to explain and drifts between runs. Breaking a judgement into smaller questions that can each be checked, and combining the answers with rules you can read and audit, makes the result consistent and defensible. Keep it simple enough that the people relying on it can explain it to someone else.

11. Once the system has decided, the assistant explains

As more of the product's answers were generated and stored, a conversational assistant that reasoned from scratch started to contradict them. The fix was a rule: the assistant explains what the system has already decided, and doesn't invent or re-rank. One source of truth across every screen matters more than a clever answer on one of them.

What I would do differently

I would automate evaluation of AI output much earlier. An automated grader doesn't need a known right answer to catch invented details, answers that stray from the context, broken formats, or regressions when a prompt changes. Only one layer truly needs a person: whether a recommendation is right for that organisation.