We have all heard the advice that generative AI agents need context to produce good content. Technical writers already use information design assets such as information models, termbases and style guides to help keep the content they produce consistent across the team.

So what happens if we give an AI author agent the same assets we give our colleagues?

Information design assets do influence the agent - for better or for worse

• It works
  With an information model, structural conformance improved from 68 to 100, on every run. Providing a termbase improved the terminology conformance by 15 to 20 points.
• Some assets matter more than others
  The information model made the biggest difference to document conformance. The termbase had a clear impact, and providing audience profiles made coverage more consistent. The style guide, however, made no measurable difference.
• Costs go up
  An author agent given the design assets used about eight times the input tokens to create a guide - about 2.0 to 2.4 million against about 0.3 million per guide.
  Writing took about 4 times as long: about 8 minutes against about 2 minutes. Although, interestingly, QA review was about 25% quicker, with a governed guide ready in about 30 minutes rather than about 40 minutes.
• The quality of the assets matters
  Judged against IBM’s characteristics of quality technical information, the governed guides scored lower than ungoverned ones. The governed guides were marked down because they provided fewer examples, scenarios and statements of purpose. The information model did not include these elements, so the author agent didn’t create them.

The information design assets

For the test, the author agent was provided with the following information design assets as guidance on how to create the user guides.

• Information model
  The document structure (the document model) describing the required sections, the topic-level structure (the content type model), and the metadata each page carries.
• Audience profiles
  Who the information is for, their roles, goals and what they already know.
• Termbase
  Approved terminology list, including the terms to avoid, and applicable audience profile.
• Style guide
  Writing style, tone of voice, and sentence-level rules.

The test

An author agent was used to produce a user guide for two open-source applications. The same agent was used for each test run, with only the information assets changing for each run.

• The author agent
  The author agent used was c8s-labs’ Dai Bach using Gemini 2.5 Flash through OpenRouter, with the same prompts and settings on each run.
• The output
  The author agent planned and wrote a complete software user guide as Markdown files. These were then compiled with Blume into a completed documentation site. The completed sites were assessed for conformance and quality by The Judges.
• The source content
  The input source was the code bases of two open-source applications: Kimai, a time tracking app, and Paperless-ngx, a document management system. Both have roles, permissions and an admin area. Their existing documentation was not provided to the author agent.
• The test ladder
  The test protocol was a simple ladder with the author agent getting one more asset per rung. There were ten runs per rung for each product, so 100 build runs in all.

The judges

To assess the quality of the user guides produced during the test, every complete build of the user guides was judged against two dimensions:

1. Conformance to the information design assets
   The QA module from c8s-labs’ Content Ops Director reviewed each guide against the full set of information design assets. The QA module uses a mix of deterministic checks, and probabilistic checks performed by an LLM judge.
2. Content quality
   Two LLM judges, Claude Opus 4 and Grok 4.6, scored each guide against IBM’s characteristics of quality technical information.

The results

There was a clear distinction between the governed and ungoverned user guides produced. The ungoverned guides averaged only about 25 pages covering about 25 subjects. However, each page was long with about seven sections on each page.

The governed guides, on the other hand, were longer, averaging about 55 pages covering 46 to 49 subjects. However, each page was shorter with only one or two sections on each page. In addition, the governed guides had about 55 internal links each, whereas the ungoverned guides only had one link across all the guides.

Conformance is the share of content units passing the QA module tests for structure, terminology, and style, out of 100.

Quality score is the mean of the IBM easy-to-understand, easy-to-find and easy-to-use scores from the Claude and Grok judges, out of 100.

Coverage is the average number of distinct subjects in a guide, counted deterministically. It is a count, not a score.

Conformance to the information design assets

• The Information model had the biggest impact
  Structural conformance went from 68 to 100 and stayed there on every subsequent run. This seems obvious as with no guidance the author agent couldn’t know what to conform to. However, the largest impact on the score was from the inclusion of the metadata the document model requires. The ungoverned guides contained no metadata, so were scored down.
• The termbase did improve terminology conformance, but not completely
  Terminology conformance improved markedly when the author agent had the termbase. However, it did still make errors about a third of the time.
• Audience profiles made coverage more consistent
  Giving the author agent context about the audience improved the consistency of the subjects covered across the guides produced. For example, Paperless-ngx user guides went from 7% commonality for the ungoverned guides up to 30% after the audience profiles were included.
• The style guide changed nothing measurable
  The style conformance was consistently high, and providing the author agent with the style guide didn’t alter that. Given that the style guide in this test followed best practice, for example using active voice and sentence-case headings, it is likely the author agent was aware of these rules.

Content quality

The governed guides scored consistently lower on the content quality benchmark. When looking more closely a picture emerged. The governed guides performed poorly against the following criteria:

• Tasks carry enough conceptual introduction
• Scenarios the reader recognises
• Why and when a task matters, not only how
• Real examples rather than abstract description
• A reader arriving mid-guide can orient themselves

The content types in the information model define the sections required for each topic. The information model used in the test is designed to be lean, and not to include examples, scenarios or extensive conceptual information. So the design assets did their job, but they were not aligned with the expectations of the quality benchmark used.

In the other direction, the information model specified shorter topics with related links between them. The governed guides scored highly on links between related topics.

The cost

The assets made creating the content more expensive, but in turn made QA cheaper.

The author agent used about eight times the input tokens to write a governed guide, and took about 9 minutes instead of under 2. However, it took about 20 minutes to check against 37 minutes for an ungoverned guide. So overall, the governed guides were completed more quickly.

Per-guide tokens average the two products.

Times are wall-clock and include time queued behind other runs for the GPU judge.

The conformance QA step uses about two and a half times the tokens on an ungoverned guide. This is likely because the longer pages hold more sections each, and the judge works section by section.

What the test does and doesn’t show

• Samples are small
  There were only 10 runs per rung per product: enough to show direction and separation, but not precise effect sizes.
• The quality scores come from two LLM judges
  A human judge or an additional calibration of the LLM judges may refine the scores.
• Coverage compares runs with each other, not with the product
  Coverage was based on the subjects within the guides. There was no separate judgement of what the ideal subject count of each product was. This is acceptable when looking at the effect of the design assets, but not for assessing the authoring agent’s planning capabilities.
• No factual correctness test
  The testing was focused on the effects of the information design assets on the author agent, not the capabilities of the author agent.
• The scope is narrow
  The test covered only software products, and user guide document-type. Results may differ for other product types or audiences.
• Only a single LLM used for the author agent
  Different LLMs may respond differently to the design assets. For example, another model may have started with a lower style conformity score.

What this means

If you are looking at how to introduce AI into your content supply chain, leverage the information design assets you already have to direct your authoring agents, and to confirm the quality of their output.