Knowledge Management Perspective
Correct and Unusable
What I learned reading procedures a language model wrote.
For part of my career I read procedures a language model had written and decided whether they were good enough to publish. Almost all of them were accurate. That turned out to be the problem.
The failure I kept finding did not look like a failure. Every sentence was true. Every step referred to something real. Hand the document to someone who already knew how to do the task and they would read it and nod.
The shape of it
A step would say: verify the account is in good standing before proceeding. True. Correct. Nothing in it is wrong. A reader who already knows where that information lives does it in four seconds. A reader who does not know has a decision to make and no information about how to make it.
Verify the account is in good standing before proceeding.What does good standing mean here. Where is it displayed. What does it look like when it is not in good standing.
The procedure has quietly handed the hardest part of the task back to the person least equipped to do it, and it has done so in a sentence that will survive every review it ever faces.
Why accuracy review never catches it
I started calling this a knowledge assumption, because that is all it is. The writer assumed the reader knew something. The writer was not a person. Accuracy review does not catch it. Accuracy review asks whether a statement is true, and this failure is made entirely of true statements. Fact-checking a document like this returns a clean report. So does a style pass. So does a subject matter expert reading it, because the subject matter expert is the one person in the building guaranteed not to notice, having loaded the missing knowledge into their head years ago.
The gap is not in the content. It is between the content and the reader, which is a place most review processes never look.
It is worth being precise about where this comes from, because it is not a defect in any particular model. Procedural writing on the internet is overwhelmingly produced by people who already know the answer. That is what these models learn from. The voice of expertise, including everything expertise leaves out, is the native voice of technical documentation as a genre, and a model trained on it reproduces that voice faithfully, omissions included.
There is also a simpler thing going on. A model has no experience of not knowing something. It cannot arrive at step four and feel the specific, slightly panicked blankness of a person who does not know what the screen is supposed to look like. It has never been stuck. Nothing in its training gives it a reason to write for someone who is. So it writes for itself.
The test that works
The test that works is not a reading test. Reading a procedure rewards fluency. A well-built sentence about a task you understand feels complete, and that feeling of completeness is exactly the signal you cannot trust here.
Following the procedure is different. Pick someone who does not know the task, hand them the document and nothing else, and watch where they stop.
They will stop in places that make no sense to anyone who knows the work, and every one of those places is a knowledge assumption made visible.
It is slow. It is also the only method I found that reliably surfaces the thing, and it works on human-written procedures too, which was its own uncomfortable discovery.
What Came Out of It
Four conditions a procedure has to meet
What came out of it was less a checklist than a set of conditions a procedure had to meet before it could reach anyone.
-
01
One action per step
Each step is one action, not a bundle of actions collapsed into a sentence because they happen together in the mind of someone fluent.
-
02
Stated criteria
Each decision point states its criteria. If the reader has to judge something, the procedure says what to judge it against.
-
03
Every branch written out
If there are two ways forward, both exist on the page, and a reader who takes the unusual path is not sent back to guess.
-
04
A named end state
Each step names the state the reader should be in when it is done. Without it, a reader who has made a mistake has no way to discover it until several steps later, when recovery is expensive.
This one mattered more than I expected. None of these are writing rules. They are structural, and they can be checked by someone who does not know the subject, which is the entire point.
Why this matters now
The reason any of this is worth writing down is that the volume has changed.
Procedures used to be expensive. Somebody had to sit with the work, understand it, and write it up, and that cost acted as a filter. Now a competent draft takes a few seconds, and the review sitting between that draft and a reader is usually an accuracy check, because accuracy is legible, fast, and easy to defend in a meeting.
A wrong procedure gets caught. Somebody follows it, something breaks, and the document is traced backward. A correct one that nobody can follow does not get caught. It becomes a support ticket, a second call, a workaround passed between colleagues, a quiet decision to stop consulting the documentation at all. None of that gets traced back to the page it came from, because the page was never wrong.
That is the failure worth building for. Not the model that lies. The model that tells the truth to somebody who is not there.