Home / Writing and talks / AI systems

ML decides, the LLM writes, code checks

Most of what gets called AI in a product is three different kinds of work. Knowing which part is which is how you keep a regulated customer's trust, and how you avoid reaching for an agent framework when a for-loop would do.

I spent some time playing with a bank's public product data, pulled through its API, to reimagine how AI could work in a regulated environment. I worked through five use cases and built three of them as working prototypes. I labelled every step by the kind of work it was, because the first question a risk team asks is which part is AI, and what happens when it is wrong. The labels turned into a rule I now apply to everything:

Machine learning decides who or what. The language model decides how it is said, never who gets it. Code checks both. A person approves.

Four kinds of work

Plain code

Data, business rules, search, calculations, checks, the interface. Deterministic and testable.

Decides what is true

Machine learning

Learns patterns from data to predict or rank, such as who is likely to respond. Measurable and explainable.

Decides who or what

Language model

Reads or writes language: messages, summaries, answers. Flexible, and the hardest part to verify.

Decides how it is said

A person

Approves, edits or rejects, and is accountable for the outcome to their own customers and board.

Decides whether it happens

Three ways to wire it

Most of the time the honest answer is plain code plus one model call. Frameworks help in two specific places: LangChain when you need standard plumbing for documents at scale, and LangGraph when the flow has to branch, loop, keep state or wait for a person.

Five banking use cases, one pattern

The same split holds across very different jobs. In four of the five, the decision is made by a model or by rules, and the language model only writes. The one that needs an orchestration framework is the one with a loop and an approval in the middle. The three I built are on their own page.

Use caseWhat decidesThe language model's jobWhat code checksIn production
Product answers for staffCode retrieves the relevant product sections (search is a scoring formula, not ML)Answers only from those sections, cites them, or refusesEvery citation points to a real section; a freshness note is addedPlain SDK. LangChain's loaders and retrievers save plumbing at scale
A copilot for relationship managersRules and a simple model rank which clients need attentionWrites talking points only from facts computed in codeEvery number and claim matches a source factLangGraph, for the redraft loop, a safe fallback and an approval that can wait hours
Campaign targetingAn uplift model picks who the offer changes behaviour for, after consent filteringWrites one variant of the offer per segmentRates, terms and banned phrases; one retry, then a templateNo framework. A scheduled ML pipeline and a loop
Next best productA propensity model ranks; eligibility rules decide what is allowedWrites a one-line opener for the relationship managerIt names the right product and invents no figuresNo framework. Batch scoring, an eligibility service and a cached API
Re-engaging dormant investorsA model picks who might return; rules pick an education-first nudgeWrites the nudgeBlocks tickers, buy or sell language and past returnsNo framework for nudges. LangGraph only for a research brief with tools and a compliance sign-off

When a framework earns its place

The copilot is the clearest case. Every brief needs the same data, so gathering it is fixed steps in code, not an agent choosing tools. What a graph adds is the loop and the pause: a failed fact-check sends the draft back at most twice, then falls back to a template flagged for review, and the whole thing waits, with its state saved, until the relationship manager approves. Every step is named and traced, which is exactly what a bank's risk review wants to see.

Questions I get asked

Retrieval or fine-tuning?

They solve different problems. Retrieval leaves the model alone and looks documents up when the question is asked, so it suits knowledge that changes, such as product terms and policies, and gives you citations, an audit trail and per-user access control.

Fine-tuning changes the model's weights, so it suits behaviour: a format, a tone, a narrow classification, or a smaller model doing one job cheaply. It costs more to update and can make the model worse at things it used to do well. Techniques like LoRA, which train small add-on weights and leave the base model frozen, make it cheaper. In a bank I default to retrieval, and fine-tune only a high-volume, narrow task.

Why not let a second model check the first?

For exact facts like balances, rates and dates, code is cheaper, repeatable, and doesn't need checking itself. I add a model as a judge only for what code can't check, such as whether a claim is supported by its source, and only after comparing its verdicts with human-labelled examples.

How do you evaluate an agent?

Check the final output, and also the path: did it call the right tools, in a sensible order, in a reasonable number of steps? Track cost and latency per run, and replay a set of recorded scenarios before every release. Every failure becomes a new test case.

What are the security risks?

Prompt injection: retrieved documents or news can carry instructions aimed at the model. So tools are read-only and allowlisted, any action that writes waits for a person, and outputs are checked. Saved state holds customer data, so it needs encryption, masking and a retention policy.

Won't loops make it slow and expensive?

Cap the retries, use a smaller model for drafting where evaluation shows it's good enough, cache what repeats, and measure the cost of each brief against the time it saves the person using it.

Workflow or agent?

If every run needs the same information, gather it in fixed steps. That is a workflow: fewer surprises, easier to test. An agent that chooses its own tools earns its place only where what needs looking up genuinely varies, such as free-form follow-up questions.

The rule underneath

Facts live in code. The model only writes. Code checks what it wrote. A person approves anything that matters. Everything else, including which framework to use, follows from where a decision sits and what it costs to get it wrong.

None of this depends on a particular model or vendor.