Skip to content
AI Architect Academy

Free. No form in front of it.

The decision record, and one decision worked all the way through.

An architecture decision record is the smallest document that makes a decision reviewable a year later by someone who was not in the room. The course is built out of them. Here is the template we use, then one record filled in for a system that many teams are being asked to build right now.

The worked example is a composite written for teaching. It is not a client, an employer, or any real organisation, and its numbers are illustrative budgets, not measurements.

The template

Seven fields. If a field is hard to fill, that is the finding.

TitleOne sentence, in the form of the decision, not the topic. “Retrieval before fine-tuning”, not “Model strategy”.
StatusProposed, accepted, superseded by ADR-n, or reversed on date. Never “in progress”.
ContextWhat is true that forces a decision now. Constraints you did not choose. The measurable failure the system exists to prevent.
DecisionWhat you will do, stated so plainly that a reviewer could check whether you did it.
Options rejectedEach serious alternative, why it lost, and under what change it would win. This field is where the judgement lives.
Cost of being wrongWhat it costs, in time, money and trust, if this decision turns out wrong, and how soon you would know.
Revisit whenThe observable trigger that reopens the decision. A date is acceptable. “When we have time” is not.

Worked example

ADR-0001: Retrieval before fine-tuning for a support-answer assistant

StatusAccepted. Revisit trigger set (below).
Decision ownerThe engineer accountable for the first release. Not a committee.

Context

A product team at a mid-size software company is asked to ship an assistant that answers customer support questions from the company’s own documentation. Leadership has already said the word “fine-tune” in two meetings. The documentation is about over a thousand pages, changes weekly, and is the only thing the assistant may cite. The measurable failure the system exists to prevent is a confident wrong answer that a customer acts on. A slow answer or an honest “I don’t know” is acceptable for the first release. A wrong one is not.

Constraints not chosen by the team: an eight-week window to a first internal release, no labelled dataset of question-and-answer pairs, one engineer full-time and one part-time, and a data-handling rule that customer conversations may not leave the company’s own environment for training.

Decision

The first release retrieves passages from the current documentation at question time and asks a general model to answer only from those passages, citing them. No model is fine-tuned in the first release. A question with no passage above the retrieval threshold gets a fixed “I can’t find that in the documentation” response and a link to a person.

Options rejected

  1. Fine-tune a model on the documentation. Rejected because there is no question-and-answer dataset, the documentation changes weekly so the model would be stale within days of every training run, and a fine-tuned model cannot cite the passage it drew from, which removes the one mechanism that lets a support agent verify an answer. It would win if the documentation were stable for months and the goal were tone or format rather than facts.
  2. Fine-tune and retrieve. Rejected for the first release because it doubles the surface to evaluate before the team has an evaluation at all. It becomes worth testing once retrieval alone has a measured wrong-answer rate to beat.
  3. Put the whole documentation in the prompt. Rejected because over a thousand pages exceeds any sensible per-question cost budget and, more importantly, long-context answers cannot be checked: the reviewer cannot tell which page produced the answer. It would win for a documentation set under a few dozen pages.
  4. Buy a hosted support-bot product. Not rejected on merit; rejected because the team could not get a written answer on where customer conversations would be stored, and the data-handling rule is not negotiable. Reopen if a vendor answers that question in writing.

Cost of being wrong

If retrieval alone cannot reach an acceptable wrong-answer rate, the team has spent roughly five of eight weeks and owns a working evaluation set of real questions, which is the input the fine-tuning option needed and did not have. The reversal cost is therefore low and the work is not wasted. The expensive failure is the one this decision was chosen to avoid: shipping without a way to check answers, and finding out from a customer.

Budgets set at decision time, to be measured against, not assumed: answer latency under eight seconds at the 95th percentile; retrieval cost per question small enough that a full day of expected traffic costs less than one hour of a support agent; and a wrong-answer rate on the evaluation set below the rate of the current human-written macro replies, which the team measures in week one before writing any assistant code.

Revisit when

  • The evaluation set reaches 300 real questions with reviewed answers, whatever the wrong-answer rate.
  • The wrong-answer rate on that set stays above the human-macro rate for two consecutive weeks.
  • The documentation change cadence drops to monthly or slower for a quarter.
  • A vendor answers the data-handling question in writing.

What this record teaches

The decision itself is unremarkable. The value is in the rejected options and the revisit triggers: they turn a preference into a position that can be checked and, if necessary, reversed cheaply. The cohort works through the sequence of these records for a system end to end, in the order you actually have to make them.

If this is obviously worth more to you than the docs, the waitlist is the place to say what you would pay for the rest of the sequence.

Join the waitlist