The Engine was the server I built so the Agent could run real checks on its own drafts. It did two jobs. It ran our content rules as Python functions, and it measured a draft against a fingerprint of the language a product had already shipped. The Agent called it before recommending any wording, then revised with what came back.
- Question
- How can we be sure an agent followed a rule?
- Problem
- A large share of our guidance had a yes or no answer, but an agent only got it as sentences in a prompt, to apply as best it could. Things slipped into the product, and nobody could guarantee a rule had been followed.
- Solution
- I had AI write those rules as Python functions and ran them on an MCP server I built, next to a fingerprint of each product’s shipped language. I kept the server warm so every draft could be checked before anyone saw it.
- Outcome
- The Agent held to our rules more consistently, with less back and forth before a line was ready. The dimensions gave us a frame of reference for discussing language that didn’t exist before.
Guidance that should have been a check
When I sorted our guidance for the Index, a large share of it turned out to have a yes or no answer. A title either fits in 100 characters or it doesn’t. A word is either on the banned list or it isn’t. Rules like these sat in prompt files as sentences, beside guidance that really did take judgment. Left as sentences, they depended on a model holding every rule in mind while it wrote, and as the list grew, things slipped through and reached the product. I don’t count that against the people who wrote the guidance. Written guidance is great for humans. It is not great for an agent we want to hold to a target.
If you’re going to ask an agent to do math, why not give it a calculator instead of asking it to do mental math? Counting characters is arithmetic, and so is looking for a banned word. It’s dirt cheap to run a Python script on a banned word list, and the agent doesn’t have to spend tokens reasoning about whether a word is on it. People’s time was part of the cost too. I put together a long explanation, with evidence, of why determinism could make us more efficient and stop wasteful waves of 50 or more content designers reviewing what a pass or fail check could settle.
So I used AI to write Python scripts that enforced every rule with a yes or no answer. A rule stopped being a sentence like “this must be less than 100 characters.” It became a hard Python rule the agent called.
Example rule
Keep the title less than 100 characters.
As a sentence in a prompt
The agent reads the rule along with every other rule. It has to remember the limit and apply it while it writes.
As a check on the server
len(title) < 100
The function counts the characters and compares the count with 100.
- ✓99 characters passes
- ✕100 characters fails
A server the agent could call
An MCP server is a program an agent can call as a tool in the middle of a conversation. MCP stands for Model Context Protocol. The agent sends a request, the server does the work, and the result comes back before the agent answers. The existing content design server retrieved guidance as data, which hands the agent a rule as text and leaves the agent to interpret and apply it. I needed a server that could also execute a check and return what it found, so I built one for the rule functions and the language measurements.
We didn’t have engineering staffed for this work. It fell to content design and our agents to figure out, and I led it. I created the server and installed the Python libraries it needed. Some were already in house, and some needed a fairly quick approval. The language libraries aren’t mine. They are open-source Python projects that other people wrote, such as spaCy, a widely used library that splits text into words and sentences, then tags each word’s part of speech and its role in the sentence. My part was the checks, the product references, and the workflow that put both in front of the Agent while it drafted.
Example: the check on one draft
The Agent’s first try at a Wi-Fi notification for the glasses
A Wi-Fi network is available nearby. Please tap to connect your glasses.
Tools the Agent called on the server
notification_lintcontent_reviewcompare_to_target
Returned to the Agent
Rule functions
- ✕Terminology. “Tap” assumes touch input. The glasses’ guidance asks for a verb that works with any input method.
- ✓Length. 72 characters. A push notification allows 240.
- ✓Privacy. Nothing flagged.
Language measurements
- ↔Politeness. Softer than notifications on the glasses usually are.
- ↔Lexical density. Fewer content words than usual.
- ·Every other dimension sat inside the product’s usual range.
Keeping it warm
I directed the Agent to send what it was about to recommend to the server before it gave text to anyone, and it sometimes checked its revision as well. That put the server in the path of every answer. Called cold, the server had real latency, because it had to start up and load its language libraries before it could check anything. So I set it up on Meta’s internal serverless infrastructure, where it always ran warm, with the server, its libraries and the product fingerprints already loaded when a call came in. The Agent could then check a suggestion and revise it inside the conversation without that start-up delay.
Checking one suggestion
Called cold
- Suggestion sent
- Start-up delayLanguage libraries load
- Check
- Revise
- Answer
Kept warm
- Suggestion sent
- Check
- Revise
- Answer
Checks and measurements
One server held two separate jobs. The first was the rule functions, the checks pulled out of the guidance. Each one took a draft and returned a finding, and they came in a few kinds.
- Pattern
- “Don’t use em dashes.” The function searches the text for the character.
- Word list
- “Never say ‘simple’ or ‘easy.’” The function looks up each word of the draft in the product’s list of banned terms.
- Limit
- “Keep the title less than 100 characters.” The function counts and compares.
- Sentence structure
- A rule against passive voice. spaCy parses the sentence, and the function looks for a passive construction in the parse.
- Reading level
- A reading grade limit. The function applies a readability formula and compares the grade with the limit.
The second job was measurement. A draft could pass every rule and still not sound like the product. It might ask for action more directly than the product usually does, or hedge more, or build longer sentences. For this, the server measured the draft along a set of linguistic dimensions and compared each reading with the same measurement taken from the product’s shipped strings. Everything went back to the Agent as one scorecard. The two kinds of result carried different weight. A rule finding was a fact about the draft, and a measurement was a reading to interpret, so the Agent could decide that a difference was justified.
Linguistic dimensions
My background in poetry gave me a way into voice. Meter always enchanted me, because you could learn the mechanics of rhythm and use them to create an effect. Guidance like “be supportive” or “be conversational” names an effect as well. It describes the experience of the person receiving the language and says nothing about how to produce it. An agent needs the how. So does the calculator it might use to check its own output.
I’m wary of “voice” and “tone” as descriptors for the same reason. Both are metaphors for what is actually happening in the language. People argue over them. So I went to linguistic research and to stylometry, the statistics of writing style, looking for the features that produce those effects. I ended up with a set of dimensions, each resting on published work: Searle on speech acts, Brown and Levinson on politeness, Biber on register, and Halliday on lexical density, among others. Each one reads a different feature of the same string.
- Speech acts
- What a sentence does: whether it tells you something or asks you to act. The scale runs from informing to asking.
- Politeness
- How a request is softened. The scale runs from direct, with few hedges, to deferential wording with hedges and softeners.
- Emotional tone
- How much feeling the words carry, from analytical to emotional.
- Register
- Whether the language reads like conversation or like information.
- Word origins (etymology)
- The share of words with Latin roots against Germanic ones. It is the best single sign of how formal something sounds.
- Concreteness
- Whether the words name things you can see and do, like “tap” and “swipe,” or abstractions like “compliance.”
- Syntax
- How sentences are built, from simple to nested.
- Lexical density
- The proportion of words that carry meaning, such as nouns and verbs.
- Rhythm (prosody)
- The pattern of stress across a line, from uniform to varied.
- Readability
- How much work the text takes to read, from dense to easy to scan.
These are different features of the same text. An error message can be easy to read and still hedge so much that it never plainly says what happened. Each dimension also asks its own question of a line like “be supportive,” and for a specific message, the answers become instructions.
Example: what “be supportive” asks of one message
The moment
Someone’s connection drops while they’re writing a message. The app keeps their draft.
- Speech acts
- Reassure first, then offer a next step.
- Politeness
- “You can” leaves the timing with the person.
- Concreteness
- Name what was saved and what happens next.
- Syntax
- Give the reassurance its own sentence.
One message that does all four
Your draft is saved. You can send it when you’re back online.
Comparing a draft with a fingerprint
The dimensions needed a reference, and that was the fingerprint. A fingerprint is a set of measurements taken from the strings a product had already shipped, with a range for each kind of situation, such as error messages or notifications. The Index article covers how one was built. When the Agent sent a draft, the server measured it the same way and set each reading against the product’s range for that situation. A reading inside the range matched how the product usually sounds there. A reading outside it told the Agent which feature was off, and in which direction. The Agent then adjusted its recommendation, sometimes with very small changes to the wording, to bring it in concert with what the product already says in that context.
Example: two options for a Horizon error message
Language measurements, against the range for Horizon error messages
Speech actsinforms → asks
Politenessdirect → hedged
Emotional toneanalytical → emotional
Registerconversational → informational
Word originsGermanic → Latinate
Concretenessabstract → concrete
Syntaxsimple → nested
Lexical densityfewer content words → more
Rhythmuniform → varied
Readabilitydense → easy to scan
Where the product’s error messages usually sitOption AOption B, the one the Agent recommends
Both options pass every rule. Only the second reads like the product’s error messages. It drops the casual interjection and the excess reassurance, and it adds a specific action. The rules alone could not have told the Agent that. Neither end of a row is better. Each row describes one feature of the language, and the question is how a draft compares with the product’s own language in that situation.
Limits of a reading
Short strings were a concrete limit. When I shortened a photo confirmation to six words, some measurements swung to extremes and others flattened, because the reference contained longer text. The weaker match didn’t mean the shorter line was worse. The Agent recognized the length problem, gave the reading less weight, and turned to the wording question that mattered.
A match with a fingerprint also proves less than it seems to. It shows that a draft sounds like what the product has shipped. Whether the draft is right for the person reading it stays a judgment call, and that call belonged to the Agent and the content designer. The measurements were not perfect instruments either. I used the same functions across the board, so the readings were consistent with each other, and whether a draft did or didn’t match along a dimension was still useful to know.
Outcome
- Back and forth
- With the server in the loop, the Agent fixed its own options before anyone saw them, and that prevented a lot of back and forth. The loop was sometimes expensive, and it was worth it.
- Consistency
- In A/B tests with and without the server, the Agent held to our rules more consistently with it. A rule check also gave the same draft the same answer every time.
- Guardrails
- I also tested agents built from a prompt file alone, and I noticed a lot of drift from their product’s targets. Against the same targets, the Agent with its checks and the product’s fingerprint was much more accurate. That gave a team an agent whose guardrails actually stay in place.
- Explaining a choice
- One of the hardest parts of content design at a big company is helping people understand the why behind a language choice. People have their own intuitions about language, and “just trust me” doesn’t always work. The dimensions gave us something to point to, such as how the product already says this or how readable a line is. That was a frame of reference for discussing language that didn’t exist before.