What You’ll Learn

Docs as Tests & AI extends the original strategy to the four places AI shows up in your documentation pipeline, and shows you how to verify each one. By the end, you’ll be able to:

• Identify the four AI integration points in your doc pipeline and what each one needs from testing
• Choose between probabilistic and deterministic verification based on cost, risk, and product cadence
• Write workflow documentation — project descriptions, agent definitions, plans, skills — that AI agents can actually execute
• Build self-healing systems that detect drift, diagnose the cause, fix the docs, verify the fix, and report what happened
• Establish governance frameworks that balance agent autonomy with the right amount of human oversight

The book also includes lessons and case studies from teams at Anthropic, Avalara, Google, Kong, and Vercel — real numbers, real tradeoffs, and real workflows you can adapt.

Who It’s For

Technical writers, documentation engineers, developer advocates, and any team using AI in their documentation workflow, whether you’re generating drafts with an LLM, feeding docs into a chatbot or RAG system, or letting agents act on your procedures. The book assumes basic familiarity with documentation work but doesn’t require any prior experience with Docs as Tests.

With forewords by Scott Abel and Michael Iantosca.

Get Your Copy Today

If your docs touch AI — or AI touches your docs — this book gives you the verification layer that makes the whole pipeline trustworthy.

Ready to make your AI-powered docs trustworthy? Get your copy today!

But what if the same LLMs that can read and summarize documentation could also write the tests for it?

That’s the idea behind AI-assisted test generation: use an LLM to convert documented procedures into executable test specifications, have a human review the output, and then run those tests deterministically from that point on. You pay the AI cost once during generation. After that, execution is deterministic. No LLM inference, no variability, no per-run API costs.

Generate once, run forever

Note where the AI sits in the workflow. The LLM is the author of the test code, not the runtime. It reads a how-to guide, interprets the steps, and produces a structured test specification. A human reviews that spec for correctness. Then a deterministic test runner executes it—no LLM involved at runtime, no variability between runs, no API costs per execution.

This is different from probabilistic testing, where an LLM evaluates docs at runtime by interpreting content and judging quality on each pass. Probabilistic approaches have their place (I cover them in Docs as Tests & AI), but they introduce variability: the same input can produce different results across runs, and you’re paying for LLM inference every time. AI-assisted test generation avoids both issues by confining the AI to a one-time generation step.

The result is a test suite that behaves exactly like hand-written tests (deterministic, repeatable, and cheaper to run) but was created in a fraction of the time.

How it works in practice

Doc Detective’s agent tools implement this pattern as a set of agent skills. The doc-detective-test skill defines a workflow that converts documentation procedures into executable test specifications:

1. Parse — Read a documentation file and identify step-by-step procedures
2. Generate — Convert each procedure into a structured test specification
3. Validate — Check that the generated spec is structurally correct (mandatory gate—the process stops here if validation fails)
4. Execute — Run the tests against the actual product
5. Analyze — Report results with failures mapped back to documentation sections
6. Fix — Optionally iterate on failing tests with confidence-based suggestions

Here’s what that looks like with a concrete example. Say you have a getting-started guide:

## Create an Account

1. Navigate to https://example.com/signup
2. Enter your email address
3. Enter a password
4. Click "Create Account"
5. Verify you see the Dashboard

The LLM reads those steps and generates a Doc Detective test specification:

{
  "tests": [
    {
      "testId": "create-account",
      "steps": [
        { "goTo": "https://example.com/signup" },
        { "type": { "keys": "test@example.com", "selector": "#email" } },
        { "type": { "keys": "SecurePass123!", "selector": "#password" } },
        { "click": "Create Account" },
        { "find": "Dashboard" }
      ]
    }
  ]
}

Each documentation step maps to a test action: navigation becomes goTo, text entry becomes type, interaction becomes click, and visual verification becomes find. The test runner (Doc Detective, in this case) executes these actions in a real browser and reports whether each step succeeded.

A critical middle step—spec validation—checks the generated spec before any test runs. The validator confirms that the tests array is well-formed, that each step contains exactly one recognized action, and that action parameters match expected types. This catches the most common LLM generation errors (hallucinated action names, malformed parameters, missing required fields) before they waste execution time.

The human in the loop

AI-generated tests aren’t automatically correct. The LLM may interpret ambiguous documentation in unexpected ways, choose the wrong selector for an element, or miss context that isn’t explicitly stated in the docs.

That’s why human review is a required step, not an optional one. After generation, a person should check:

• Action mapping — Did the LLM correctly interpret each documentation step? A step that says “select your region” might map to a dropdown select, a radio button click, or a text input depending on the UI.
• Selectors and identifiers — When the LLM couldn’t use plain text matching (like { "click": "Submit" }), did it choose reasonable selectors that match your product? CSS selectors are brittle and should be a last resort.
• Missing steps — Did the LLM capture implicit steps that the documentation assumes? Some guides assume the user is already signed in or has already configured a prerequisite.
• Test data — Are the test values reasonable? Email addresses, passwords, and other inputs should be appropriate for a test environment, and any secrets should be stored as environment variables.

The doc-detective-test skill includes a confidence scoring system for its fix loop: when a test fails and the agent proposes a fix, it assigns a confidence score (0–100). Fixes above a configurable threshold are applied automatically, while fixes below it are shown to the user for approval. This applies to initial generation, too. Outputs are drafts that needs review, not a finished product.

This is where the economics work out. Reviewing a test spec generated from a clean procedure takes minutes. Writing one from scratch takes significantly longer, especially for complex multi-step procedures. Multiply that across hundreds of guides, and the time savings are substantial.

Bootstrapping test coverage

For teams with an existing docs site and zero test coverage, the doc-detective-init skill provides a bootstrapping workflow:

1. Detect — Scan the repository for documentation files (Markdown, MDX, AsciiDoc, reStructuredText, HTML, DITA)
2. Configure — Generate a minimal Doc Detective configuration
3. Generate — Create test specifications for all identified procedures
4. Execute — Run all generated tests
5. Fix — Iterate on failures with confidence-based suggestions

This takes a repository with no test infrastructure and produces a working test suite. The tests live alongside the docs in version control, run through the same CI pipelines, and produce the same pass/fail signals as hand-written tests.

The approach isn’t all-or-nothing. You can start with one guide, review the generated tests, and expand from there. Each generated spec is an independent file that can be reviewed, edited, and committed on its own timeline.

Injecting tests for maintainability

So now you have a pile of generated test specs—separate JSON files sitting alongside your docs. That works, but it creates a maintenance problem I wrote about in Docs as Tests because when tests live in separate files, they drift from the documentation they validate. Someone updates a procedure, forgets to update the corresponding spec, and now your tests are checking stale claims. The more specs you generate, the faster this drift compounds.

The solution is to take verified test specs and embedding them as inline comments directly in your documentation source files. The doc-detective-inject skill handles this. After tests pass validation and execution, it matches each test step to the documentation step it validates and inserts it as a comment:

## Create an Account

<!-- test {"testId":"create-account", "detectSteps": false} -->
1. Navigate to https://example.com/signup
<!-- step {"goTo":"https://example.com/signup"} -->
2. Enter your email address
<!-- step {"type":{"keys":"test@example.com","selector":"#email"}} -->
3. Enter a password
<!-- step {"type":{"keys":"SecurePass123!","selector":"#password"}} -->
4. Click "Create Account"
<!-- step {"click":"Create Account"} -->
5. Verify you see the Dashboard
<!-- step {"find":"Dashboard"} -->
<!-- test end -->

Readers see normal documentation (the comments don’t render), but people or agents editing the file see the test assertions right next to the steps they check. When someone rewrites step 4 to say “Click Sign Up” instead of “Click Create Account,” the inline test comment is staring right at them. The coupling is proximity-based, not just conceptual.

This matters more at scale. A team with three hundred guides probably can’t keep separate docs and tssts in sync without heavy automation or process, and process is fragile. Inline tests eliminate the synchronization problem entirely because the test is the documentation file. One file to update, one file to review, one file to commit. Tests detected directly from documentation syntax is even better, but that’s a different discussion.

Injection also changes what CI catches. With separate spec files, a renamed button breaks the test, and the failure points to the spec file. The writer has to trace back to figure out which doc needs attention. With inline tests, the failure points directly to the documentation line where the assertion lives. The feedback loop is shorter.

You don’t have to inject every test. Some teams inject tests for critical procedures and keep separate specs for broader coverage or for content formats that don’t support comments. The point is to co-locate tests with content wherever you can, because proximity is the cheapest form of maintenance discipline.

Fitting into CI

Once tests are generated and reviewed—whether they live as separate spec files or as injected inline comments. They’re just files in your repository, and they run the same way hand-written tests do:

npx doc-detective --input docs/ --output .doc-detective/results/

In a CI pipeline, this means every pull request that changes documentation can automatically verify that the documented procedures still work. The tests are deterministic, so the same input produces the same output every time with no hard-to-forecast LLM costs per run.

This is the same pattern Kong uses for their docs. Their copy-paste-down-the-page design makes tests detectable from docs content. AI-assisted generation automates that detection step.

The feedback loop is fast: change docs → tests run → failures point to specific documentation steps that no longer match the product. Whether those tests were written by a person or generated by an AI doesn’t matter to the CI system. What matters is that they exist and they run.

Where this leads

AI-assisted test generation is one piece of a broader pipeline. The generation step uses an LLM. The execution step is deterministic. Future steps, including self-healing docs systems that automatically fix failures and test coverage analysis that identifies untested procedures, build on the same foundation.

The skills that power this workflow follow the design principles I wrote about on instructionmanuel.com: single responsibility, testable in isolation, documented interface, composable. The generation, validation, execution, and injection steps are separate skills that compose into a larger workflow. Each skill can be validated independently using deterministic quality checks.

You don’t need to hand-write every test. Use an LLM to generate the first draft, review it, then let deterministic execution handle the rest. The tests themselves are ordinary files that are readable, editable, and version-controlled but happen to have been authored faster than a person could write them from scratch.

“We’re able to answer 91% of questions with a certain answer.”

— Diana Breza, Technical Writer at Kong

About Kong

Kong is the AI and API connectivity company that enables you to build, run, discover, govern, and monetize all of your AI and API connectivity. . With multiple products spanning cloud platforms, on-premises deployments, and developer tooling, Kong’s docs team is challenged to keep their technical content accurate across a constantly evolving product portfolio.

The docs team consists of approximately six technical writers plus Fabian Rodriguez, a software engineer who focuses on docs tooling and infrastructure. Diana Breza presented Kong’s docs testing approach at Write the Docs Berlin, showcasing how their team transformed docs quality through automated testing.

The Challenge: Broken Commands and User Frustration

Before rebuilding their how-to guides, Kong’s team faced a common but painful problem: users would copy commands from the docs, paste them into their terminals, and nothing would work.

“One of the most important pieces of feedback that we got for the old site was that people tried stuff and it didn’t work. They’d copy commands or whatever and those didn’t work.”

— Fabian Rodriguez, Software Engineer at Kong

This wasn’t just frustrating for users. It was nearly impossible for the docs team to manage. Kong has multiple products, frequent releases, third-party integrations, and various deployment options. Keeping track of which commands still worked and which procedures needed updates would be overwhelming for a team of any size, and it certainly was for them.

Manually testing docs before each release simply didn’t scale.

The Solution: Tests Derived Directly from Docs

Kong’s approach to doc testing differs from what many other organizations have attempted. Instead of creating a parallel test suite that mirrors the docs, which introduces its own drift and maintenance burden, they built tooling that derives tests directly from the docs content itself.

Copy-Paste-Down-the-Page Design

Kong’s approach centered on a single docs design principle: every how-to guide should be completable by copying and pasting commands sequentially down the page. This means each guide includes all prerequisites, all necessary setup steps, all commands, and a verification step at the end.

This design serves two purposes. First, it creates a better user experience. During user testing with internal employees, Kong discovered something remarkable:

“When we would show them a how-to doc, they’d scroll up and down the page trying to read it. When we told them you can copy and paste down the page, it’s like this light bulb moment. So few docs have that. Once they learned they could do that, they just took off down the page and within several minutes they were done. Whereas normally it would have taken them an hour maybe with all the different options you could configure.”

— Diana Breza

Second, this design makes automated testing straightforward. If a human can complete a guide by copying commands sequentially, so can a machine.

Structured Content with YAML Blocks

Kong’s docs use Markdown files with YAML-based code blocks that define the instructions. Writers specify metadata like which product versions a guide applies to, what prerequisites are needed, and what entities must be created. The platform then generates both the user-facing docs and the machine-readable test instructions from the same source.

This approach eliminates a common failure mode: docs and tests drifting apart. When the source is the same, they can’t diverge. If the docs are wrong, the test fails. If the test passes, the docs work.

Automated Daily Testing

Kong runs their doc tests daily through GitHub Actions. The test runner launches a headless browser, navigates to each how-to guide, extracts the code blocks, spins up the necessary Docker containers, and executes each command in sequence. Validation steps confirm that API calls return expected responses and that the documented workflow produces the promised results.

When tests fail, the system generates an instruction file, a minified version of the guide showing exactly what commands were executed. This makes debugging straightforward and allows the team to share reproducible failure cases with product engineering.

Influencing Product Development

One unexpected outcome: Kong’s docs requirements actually influenced product development. The team wanted users to be able to configure the product entirely through copy-paste commands without creating intermediate files. This required changes to Kong’s CLI tool, decK.

“We wanted the experience to be really smooth. A person who works on decK actually changed the product so that we could have users just copy and paste down the page.”

— Fabian Rodriguez

When docs requirements drive product changes, the product becomes more user-friendly by design.

Results: Catching Issues Before Users Do

Kong’s doc testing has delivered measurable improvements across several dimensions.

Catching Third-Party Breaking Changes

Kong’s products integrate with numerous third-party services, and those services change without notice. The doc tests caught a breaking change when Keycloak modified how their Docker container runs, a change that would have broken Kong’s authentication integration guides for users.

“Out of the blue they changed the way you run the Docker container. We actually noticed that because of the tests, which was really cool.”

— Fabian Rodriguez

Part of the Product Release Process

Kong’s doc test suite has become an official part of their product release process. The team can run tests against pre-release versions of Kong products, catching breaking changes before they reach users.

When Kong launched Event Gateway, a new product, the doc tests were ready from day one. During the week before general availability, as the engineering team made last-minute API changes, the doc tests caught issues in near real-time.

“We were able to catch a lot of stuff because of the tests. They made a change to the API, they made a change to this other thing. We were able to catch those because of the tests. Otherwise we would have to spend a lot of time testing things manually.”

— Fabian Rodriguez

Doc Tests That Compliment Engineering Tests

Perhaps the most striking validation came from Kong’s engineering teams. Doc tests that simulate real user workflows often catch issues that traditional unit and integration tests miss.

This makes sense when you consider what doc tests actually validate. While engineering tests verify that individual pieces of functionality work correctly, doc tests verify that end-to-end user workflows succeed, which is ultimately what matters for user experience.

The Writers’ Experience

Adopting this approach required writers to learn new skills, particularly working with YAML syntax, but the benefits quickly became apparent.

“It was a bit of a learning curve to do the YAML, but it’s so nice because I remember pasting in a curl example and then fighting with the formatting. With the YAML you don’t have to fight the formatting at all. It also forces us to test everything. As I’m writing it, I’m also testing everything.”

— Diana Breza

The testing workflow also changed how writers interact with their product teams. When writers discover issues during docs development, they have immediate evidence and reproducible test cases to share.

“Our writers are a lot more active in project channels and team channels saying, ‘Hey, this is broken.’ Sometimes it’s like, ‘Oh, you’re right. I’ll file a bug for it.’”

— Diana Breza

This shift positions technical writers as quality partners rather than downstream dependencies for product changes.

AI is Why Accuracy Matters More Than Ever

The 91% confident answer rate from Kong’s AI chatbot isn’t a coincidence. When docs are accurate, complete, and structured consistently, AI systems can retrieve and synthesize information more effectively.

This creates a virtuous cycle: doc testing guarantees accuracy, accurate docs enable confident AI responses, and confident AI responses reduce support burden and improve user satisfaction.

As organizations increasingly rely on AI-powered tools that consume docs, the importance of docs accuracy multiplies. A single incorrect procedure doesn’t just frustrate one user who reads it; it propagates through every AI-generated response that draws from that source.

What We Can Learn From Kong

Kong’s experience shares several insights for teams considering doc testing:

• Design for testability from the start. Kong’s copy-paste approach creates better user experiences AND makes testing straightforward.

• Derive tests from docs, not parallel artifacts. When tests come from the same source as docs, they can’t drift apart.

• Doc testing can improve products. When docs requirements influence product design, everyone benefits.

• End-to-end workflow tests catch what unit tests miss. Doc tests validate the actual user journey, not just individual components.

• AI accuracy depends on docs accuracy. As AI consumes your docs, testing becomes even more critical.

Getting Started with Docs as Tests

You don’t need to build custom tooling to start testing your docs. The Docs as Tests approach works with a variety of tools, from purpose-built solutions like Doc Detective to general-purpose testing frameworks like Playwright or Cypress adapted for docs workflows.

It isn’t about which tool to use. It’s about treating docs as testable assertions about product behavior. When you do, you can verify accuracy automatically, catch regressions proactively, and build confidence that what you’ve written actually works.

Start small. Pick one high-value procedure, like your getting started guide or most-viewed how-to, and make it testable. Learn what works for your content, your products, and your team. Then expand from there.

Your users, and your AI chatbot, will thank you.

This case study is based on an interview with Diana Breza and Fabian Rodriguez of Kong. Diana presented Kong’s doc testing approach at Write the Docs Berlin.

Learn more about Docs as Tests at docsastests.com and explore Doc Detective at doc-detective.com.

I’m thrilled to announce that my book, “Docs as Tests: A Strategy for Resilient Technical Documentation” is now available! After years of developing this approach at companies like Skyflow and sharing it with the documentation community, I’ve created a comprehensive guide to help technical writers and developers keep documentation accurate and in sync with products.

Get your copy today!

What You’ll Learn

Docs as Tests introduces a powerful strategy that treats documentation as testable assertions about how your product works. By implementing automated testing for your documentation, you can:

• Catch documentation issues before your users do
• Maintain accurate screenshots, API examples, and code snippets
• Build user trust through consistently reliable documentation
• Reduce support costs and improve user experience

The book covers testing approaches for all major product interfaces:

• GUIs (web and native applications)
• APIs (REST, GraphQL, AsyncAPI)
• Command-line interfaces
• Code examples and SDK documentation

Watch the Launch Event

Missed the book launch event? You can watch the panel to hear insights from experts who’ve implemented Docs as Tests in their organizations.

Interested in a Docs as Tests Course?

Based on the enthusiastic response to the book, I’m considering developing a hands-on course that will walk you through implementing Docs as Tests in your organization. If you’re interested, please fill out this form to help gauge interest and shape the content.

Get Your Copy Today

Whether you’re a technical writer, developer, documentation manager, or anyone else involved in creating technical documentation, “Docs as Tests” will help you build more resilient, trustworthy docs.

Ready to transform your documentation strategy? Get your copy today!

August 10, 2024: Updated the tutorial to use the Doc Detective GitHub Action.

Adding Doc Detective into a CI/CD pipeline to run tests whenever changes occur in a repository is easy and gives you a flexible and powerful tech doc toolchain. In this guide, we’ll walk through setting up a basic Docs as Code pipeline using the venerable Jekyll static site generator, set up some simple doc detective tests, and put together scripts for a CI/CD pipeline using Github Actions to demonstrate how to automatically test your docs off of each update.

Prerequisites

This tutorial is written for Unix-based systems. If you’re using Windows, most of what we’re covering regarding GitHub Actions should still apply, but you may need to use different commands to get Jekyll up and running.

This tutorial is mostly focused on setting up Doc Detective in CI/CD, so we’re going to assume you have a few things already set up:

• A GitHub account
• Ruby installed as outlined
• Jekyll installed as a Ruby Gem
• Bundler installed as a Ruby Gem

The Lazy Way

If you don’t want to go through all of the following steps, you can fork the doc-detective-ci-cd starting point branch. This branch contains the files and code you would get from following the instructions in this post.

Download the contents of the branch, add them to your own repository’s gh-pages branch, and push them to origin.

git clone --branch gh-pages https://github.com/nikomancy/doc-detective-ci-cd.git

Setting up Jekyll

Let’s sprint through setting a Jekyll site up with GitHub Pages. (If you want to see a full tutorial on setting up a Jekyll site, Github has an excellent one)

1. Create a new repository on GitHub. You can name it whatever you want, but for this tutorial, we’ll call it doc-detective-ci-cd.
2. Clone the repository to your local machine.
3. Open the directory in your terminal and checkout a new branch that we’ll call gh-pages by using git checkout --orphan gh-pages.
4. Jekyll only generates the files for a new site in an empty directory, so delete all the files in the directory and run the following command:

jekyll new --skip-bundle .

This creates a new Jekyll site in the current directory. The --skip-bundle flag tells Jekyll to skip installing the dependencies, as we’ll be using Bundler to manage them.

In the root of your newly scaffolded Jekyll site, find and open your Gemfile, then comment out the line that starts with gem "jekyll",

Next, uncomment the line that starts with gem "github-pages", and save the file.

Edit that line to include the current version of GitHub Pages. You can find the current version by visiting the GitHub Pages Dependency Versions page.

gem "github-pages", "~> GITHUB-PAGES-VERSION", group: :jekyll_plugins

Save the file and close it.

Now, open the _config.yml file and change the url and baseurl to match your repository name. For example, if your repository is called doc-detective-ci-cd, you would change the url and baseurl to:

baseurl: "/doc-detective-ci-cd/" # the subpath of your site, e.g. /blog
url: "http://<your username>.github.io" # the base hostname & protocol for your site

Close the file and save it.

Finally, run the following command in your terminal:

bundle add webrick
bundle install

This will add the missing webrick dependency to your gemfile and install the dependencies for your Jekyll site.

Save and commit the changes from this section to your repository then push this branch back to origin with git push origin gh-pages.

Setting up GitHub pages

Since Github automatically looks for a gh-pages branch in your repository, you don’t need to do anything else to set up GitHub Pages. You can now visit https://<your-github-username>.github.io/<your-repository-name> to see your site.

Adding content to your site

You can add content to your site by adding markdown files to the _posts directory. In this case, we’re going to add a Markdown file that contains inline Doc Detective tests.

Create a new file in the _posts directory called 2024-02-16-reqres-docs.md, and add the following content:

---
layout: post
title:  "Reqres API"
---

## Resource Endpoint
If you execute this command in your terminal:

[comment]: # (test start {"id": "Test 1", "description": "Test if the endpoint is working at all."})

    ```sh
    curl -X 'GET' \
      'https://reqres.in/api/{resource}' \
      -H 'accept: application/json'
    ```

[comment]: # (step {"action": "httpRequest", "url": "https://reqres.in/api/{resource}"})

The API will return a list of resources.

[comment]: # (test end)

## Creating Users
If you execute this command in your terminal:

[comment]: # (test start {"id": "Test 2", "description": "Test the user creation endpoint"})

    ```sh
    curl --request POST \
      --url https://reqres.in/api/users \
      --header 'Content-Type: application/json' \
      --header 'accept: application/json' \
      --data '{
    "name": "Doc Brown",
    "job": "Test master"
    }'
    ```

[comment]: # (step { "action": "httpRequest", "url": "https://reqres.in/api/users", "method": "post", "requestData": { "name": "Doc Brow", "job": "Test master" }, "responseData": { "name": "Doc Brown" }, "statusCodes": [200, 201] })

The API will return confirmation that the request created a new user and will list the fields on that user.

[comment]: # (test end)

Now, save and commit the changes to your repository and push them to the gh-pages branch.

Setting up GitHub Actions

Now that we have a Jekyll site set up and hosted on GitHub Pages, we can set up the Doc Detective GitHub Action to run the tests whenever we push new content to the gh-pages branch.

Create a new file in the .github/workflows directory called docs-as-tests.yml, and add the following content:

name: Docs as Tests

Next, let’s add a trigger to the workflow that prompts the action to run whenever we push new content to the gh-pages branch.

on:
  push:
    branches:
      - gh-pages

Now, let’s add a job to the workflow that runs the tests. Github Actions run on a fresh virtual machine each time they’re triggered, so we need to check out the repository with our changes.

jobs:
  run-tests:
    steps:

    - name: Checkout doc-detective-ci-cd repository
      uses: actions/checkout@v4

Finally, we can run the tests by running Doc Detective and passing the path to the _posts directory in the repository we’re running the tests on. We’ll also set exit_on_fail to true to throw an error if there are any failed tests.

    - name: Run Doc Detective tests
      uses: doc-detective/github-action@v1
      with:
        input: _posts/
        exit_on_fail: true

Putting the entire workflow together, we get the following:

name: Docs as Tests

on:
  push:
    branches:
      - gh-pages

jobs:
  run-tests:
    runs-on: ubuntu-latest
    steps:

    - name: Checkout doc-detective-ci-cd repository
      uses: actions/checkout@v4

    - name: Run Doc Detective tests
      uses: doc-detective/github-action@v1
      with:
        input: _posts/
        exit-_on_fail: true

Save and commit the changes to your repository and push them to the gh-pages branch. Now, whenever you push new content to the gh-pages branch, the tests will run and you will be able to see the results in the Actions tab of your repository.

There’s a failing test in the markdown test earlier in this post. The Github action should be failing and will provide some additional details to help with fixing it. In this case, it’s just a typo in a test. But, if the service the tests rely on breaks, you will get the same error.

Conclusion

In this tutorial, we set up a Jekyll site with GitHub Pages, added content to the site, and set up a GitHub Action to run Doc Detective tests on the content whenever we push new content to the gh-pages branch. Now, as your product evolves, you can be confident that changes that break the test will come to your attention immediately.

About the tool

Doc Detective is an open-source documentation testing framework. Doc Detective has multiple components to integrate with your workflows as you need it to:

• Doc Detective: A standalone tool to test docs against product UIs and APIs.
• Doc Detective Companion: A browser extension for Chrome and Firefox to help you build tests and find CSS selectors.
• Doc Detective Core: An NPM package that provides Doc Detective’s testing functionality.

Doc Detective is open to contributions, and you can join the chat on Discord. If you like what you see, you can ⭐ star the repositories on GitHub or ❤️ sponsor development via Open Collective or GitHub Sponsors.

Thanks for reading. If you like what you read here, sign up for the newsletter to stay up to date on Docs as Tests discussions, share this article with those you think would appreciate it, or sound off in the comments below.

Azure’s Innovation Engine, introduced at the Linux Foundation Open Source Summit North America 2024, transforms Markdown into executable scripts, enabling interactive tutorials, automated tests, response validation, and execution of CLI commands. While Innovation Engine is still in the early stages of development, it shows promise for improving the quality and accuracy of CLI-based docs.

Here, we’ll cover how to use Innovation Engine to test all the bash commands in code blocks within a tutorial written in Markdown. We’ll use a simple tutorial that interacts with the ReqRes API, a fake online REST API designed for testing and prototyping. One item to note, though: Innovation Engine is bash-only at the moment, so you’ll need a macOS or Linux environment (Windows Subsystem for Linux (WSL) on Windows works great) to follow along.

Prerequisites

This tutorial is slightly more advanced than other recent ones. Let’s make sure you’re ready to go. You need

• A bash shell.
• Go installed on your machine.

Set up Innovation Engine

To install and build Innovation Engine, you need to download or build the ie binary.

On Linux, you can download the binary and make it executable with the following commands:

VERSION="latest"
wget -q -O ie https://github.com/Azure/InnovationEngine/releases/download/$VERSION/ie
chmod +x ie

On macOS (or if you like to go the hard road on Linux), you need to clone the repo and run the build command:

git clone https://github.com/Azure/InnovationEngine;
cd InnovationEngine;
make build-ie;
cd ./bin

Sample document and formatting

Here’s a simple tutorial in Markdown that uses ReqRes, a testing API that uses fake data. The tutorial focuses on two common API requests: fetching a list of users and creating a new user.

Each command is in a bash code block, and the expected output is in a JSON code block below it. By default, Innovation Engine runs commands in bash code blocks to make sure they run as expected. For example:

    ```bash
    curl -X POST https://reqres.in/api/users \
      -H "Content-Type: application/json" \
      -d '{
        "name": "Jane Doe",
        "job": "Developer"
      }'
    ```

To validate responses, you need to add an HTML-style comment before the immediately following code block and include the expected_similarity key and a value between 0 and 1. This value represents the similarity between the expected output and the actual output, allowing for fuzzy matching.

    <!--expected_similarity=0.8-->
    ```
    {"name":"Jane Doe","job":"Developer","id":"557","createdAt":"2024-07-04T21:58:42.684Z"}
    ```

Note: Response code blocks can’t have language tags, so you can’t use json or bash here. If there’s a language tag, Innovation Engine won’t consider it a response block.

Testing the commands

Using Innovation Engine, we can test the commands in the code blocks to make sure they run correctly.

1. Download reqres_tutorial.md into the same directory at the ie binary.

2. Run the following command to test the commands in the tutorial:

    ./ie test reqres_tutorial.md
   

   This command reads the Markdown file, executes the curl commands, and validates the output.

   The output should look similar to this:

    $ ./ie test reqres_tutorial.md
    bash:$ curl https://reqres.in/api/users
   
    {"page":1,"per_page":6,"total":12,"total_pages":2,"data":[{"id":1,"email":"george.bluth@reqres.in","first_name":"George","last_name":"Bluth","avatar":"https://reqres.in/img/faces/1-image.jpg"},{"id":2,"email":"janet.weaver@reqres.in","first_name":"Janet","last_name":"Weaver","avatar":"https://reqres.in/img/faces/2-image.jpg"},{"id":3,"email":"emma.wong@reqres.in","first_name":"Emma","last_name":"Wong","avatar":"https://reqres.in/img/faces/3-image.jpg"},{"id":4,"email":"eve.holt@reqres.in","first_name":"Eve","last_name":"Holt","avatar":"https://reqres.in/img/faces/4-image.jpg"},{"id":5,"email":"charles.morris@reqres.in","first_name":"Charles","last_name":"Morris","avatar":"https://reqres.in/img/faces/5-image.jpg"},{"id":6,"email":"tracey.ramos@reqres.in","first_name":"Tracey","last_name":"Ramos","avatar":"https://reqres.in/img/faces/6-image.jpg"}],"support":{"url":"https://reqres.in/#support-heading","text":"To keep ReqRes free, contributions towards server costs are appreciated!"}}
    bash:$ curl -X POST https://reqres.in/api/users \
      -H "Content-Type: application/json" \
      -d '{
        "name": "Jane Doe",
        "job": "Developer"
      }'
   
    {"name":"Jane Doe","job":"Developer","id":"334","createdAt":"2024-07-04T22:17:31.936Z"}
   

   The output shows the commands being executed and the expected output. If the actual output matches the expected output within the similarity threshold, the test passes. If the output isn’t similar enough to the expected output, the test fails.

Wrapping up

By validating your documentation with Innovation Engine, you can make sure that your code examples are always accurate and functional. While Innovation Engine is limited to testing CLI docs, it can be integrated into your CI/CD pipeline to automate your testing and make sure that your most important CLI-based docs, like installation instructions, are always up-to-date and accurate.

About the tool

Azure’s Innovation Engine is a CLI tool that transforms markdown documentation into executable scripts. This allows for interactive tutorials, automated tests, and execution of CLI commands, making sure your documentation remains accurate and up-to-date. For more information, see the Innovation Engine GitHub repository.

Thanks for reading. If you like what you read here, sign up for the newsletter to stay up to date on Docs as Tests discussions, share this article with those you think would appreciate it, or sound off in the comments below.

Today, I’ll show you how to validate a UI-based procedure with Cypress—an engineering-focused testing tool—to make sure that it works as expected. Cypress tests are written in JavaScript or TypeScript and use JavaScript-style test assertions, so if you’re comfortable with those, you’re good to go. If not, don’t worry! We’ll walk through the basics of writing a Cypress test to validate our procedure, and I’ll show you how to run the test in the Cypress Test Runner and from the command line.

Considerations

Before we get started, there are a few things to consider when using Cypress for Docs as Tests:

• Cypress is a UI testing tool. It’s great for validating UI-based procedures, but it’s not great for validating non-UI-based procedures. For example, you can’t use Cypress to validate a command-line procedure without extensions.
• Cypress tests are written in JavaScript or TypeScript. If you’re not comfortable with JavaScript or TypeScript, you might find it difficult to write and maintain Cypress tests.
• Cypress tests are separate from your content. You can’t embed Cypress tests in your content, so you have to maintain them separately. This can make it difficult to keep your tests in sync with your content, especially if you’re iterating on your content frequently.

With these considerations in mind, let’s get started!

The setup

For this tutorial, assume you have a simple procedure about searching for American Shorthair kittens:

To search for American Shorthair kittens,

1. Go to [Google](https://www.google.com).
2. In the search bar, enter "American Shorthair kittens", then press Enter.

![Search results](search-results.png)

We’ll call this kitten-search.md. Here’s a rendered preview:

Notice that we don’t have an image of the results. We can capture that as part of the testing.

Following the Docs as Tests ideal that each doc is a test spec, each procedure a test case, and each step an assertion, we can break the procedure into the following testable statements:

1. Users should be able to navigate to www.google.com.
2. There is a search bar.
3. After entering text into the search bar and pressing Enter, search results appear.

Additionally, there is the screenshot that we want to automatically capture.

To do all of this, we’ll create and run a Cypress test that steps through each of the action. First though, let’s get Cypress ready on your machine.

Install Cypress

If you already have Cypress installed, skip to the next section.

Before you can get Cypress running, you need the prerequisites:

• Node.js. I like using Node Version Manager, or you can use the official installers.

Once you have those installed, you can install Cypress with NPM.

1. Open your terminal and run the following command to install Cypress on your machine:

    npm install cypress
    npx cypress open
   

   The Cypress Test Runner opens.

2. Click E2E Testing, click Chrome, then click Start E2E Testing in Chrome. This opens a new browser window with Cypress running.

Write the test

Cypress tests are written in JavaScript or TypeScript—we’ll use JavaScript for simplicity. Here’s a test that validates the procedure we wrote earlier. Create a new file called kitten-search.cy.js in the cypress/e2e directory of your project, and add the following code:

// kitten-search.cy.js
describe('Google Search', () => {
  it('Navigates to Google', () => {
      cy.visit('https://www.google.com');
      cy.url().should('include', 'google.com');
      cy.get('[name="q"]').should('be.visible');
      cy.get('[name="q"]').type('American Shorthair kittens{enter}');
      cy.get('#center_col').should('be.visible');
      cy.screenshot('search-results', {capture: 'viewport'});
  });
});

Let’s break down each portion of the test:

1. describe and it are Mocha functions that help organize your tests. describe is a suite of tests, and it is an individual test.
2. cy.visit navigates to the specified URL.
3. cy.url().should('include', 'google.com') checks that the URL includes “google.com”.
4. cy.get('[name="q"]').should('be.visible') checks that the search bar is visible.
5. cy.get('[name="q"]').type('American Shorthair kittens{enter}') types “American Shorthair kittens” into the search bar and presses Enter.
6. cy.get('#center_col').should('be.visible') checks that the search results are visible.
7. cy.screenshot('search-results', {capture: 'viewport'}) captures a screenshot of the viewport and saves it as cypress/screenshots/search-results.png.

Note: With Cypress, you have to write your tests in the correct directory or Cypress won’t recognize them. Additionally, you can’t embed them in your content, so you have to maintain them separately from your docs. This can make it difficult to keep your tests in sync with your content, especially if you’re iterating on your content frequently.

Now let’s run the test!

Run the test in the Cypress Test Runner

In the Cypress Test Runner,

1. Click Specs in the side navigation.
2. Find and click kitten-search.cy.js.

Cypress opens and runs the test in your browser. You’ll see the test steps in the left pane and the browser window in the right pane. When the test is done, you’ll see a green checkmark next to the test name, indicating that it passed, and you’ll find a new screenshot in the cypress/screenshots directory.

If you hover over the steps in the left pane, you can see the screenshots that Cypress took at each step. This is a great way to visually validate that your test is working as expected.

Move the new screenshot into the same directory as kitten-search.md, and the procedure now renders like this:

Congratulations, you ran your first Cypress test!

Run the test from the command line

You can also run your tests from the command line. This is useful for running your tests in continuous integration (CI), or for running your tests in a headless browser.

In your terminal, run the following command to run your test in a headless browser:

npx cypress run --spec "cypress/e2e/kitten-search.cy.js"

Cypress runs your test in a headless browser and outputs the results to the terminal. You’ll see a summary of the test results, including the number of tests that passed and failed, and the time it took to run the tests.

What else can it do?

Cypress is a powerful tool that can do a lot more than what we’ve covered here, as long as your comfortable with JavaScript or TypeScript. For example, you can:

• Run your tests in CI to validate your docs on every commit.
• Use custom commands to abstract common actions and make your tests more readable.
• Use plugins to extend Cypress’s functionality.

You can also use Cypress to validate other UI-based procedures, like filling out forms, navigating a website, or interacting with a web app. Cypress is a great tool for validating UI-based procedures, and it’s a great addition to your Docs as Tests strategy.

About the tool

Cypress is a popular end-to-end testing framework that’s easy to use and has a great community. If you’re familiar with JavaScript or TypeScript, it’s a great choice for testing your documentation against your product UIs and APIs. For more information, see the Cypress documentation.

Thanks for reading. If you like what you read here, sign up for the newsletter to stay up to date on Docs as Tests discussions, share this article with those you think would appreciate it, or sound off in the comments below.

In practice, this process means defining and running automated tests based on what the documentation says about the product. Ideally, these tests should run every time updates to the product’s code is updated. However, updates to the code often occur in different environments, such as development, staging, and production. Each of these environments can benefit from adding doc-based tests, but each environment benefits in different ways.

Using Docs as Tests across development environments

How you apply Docs as Tests varies significantly across different stages of the development lifecycle, and each stage has unique advantages and challenges. Let’s delve into how this strategy plays out in development, staging, and production environments, outlining the potential benefits and pitfalls within each.

Development environment

The dev branch is a churning sea of iteration, experimentation, and changes that are checking if an idea will work and figuring out how to make it work well later. As the name of this environment suggests, it’s mainly the environment where developers work, and so most of the advantages Docs as Tests deliver here will be ones focused on developers.

Advantages

1. Early error detection: Implementing Docs as Tests in the development phase allows for the early identification of inconsistencies between the product and its documentation. Even if developers ignore these inconsistencies until they complete development, letting developers know early helps get this information to content teams sooner than later.
2. Immediate feedback loop: Developers receive real-time feedback on their changes, ensuring that new features or modifications align with documented expectations from the outset.
3. Documentation quality: Engaging with documentation during development ensures it evolves alongside the product, fostering a higher standard of accuracy and comprehensiveness.

Pitfalls

When it’s implemented correctly, Docs as Tests shouldn’t be disruptive to development. However, there are some pitfalls that teams may fall into:

1. Test creation overhead: Crafting testable documentation in a dynamic development environment demands significant effort and foresight, potentially slowing down the innovation process.
2. Maintenance burden: The fluid nature of development means documentation and tests must be continually updated, imposing an additional maintenance load on teams.
3. Risk of disruption: Introducing rigorous Docs as Tests requirements during development may hinder rapid prototyping and experimentation, stifling creativity.

The goal for using Docs as Tests in a dev environment is to help developers know when they make a change that affects the accuracy of the docs without slowing down their development. The main way to avoid this pitfall is to have broken tests warn developers rather than serve as a gate on pushing their code to the development environment. If developers see that their updates are causing a test to fail and they’re getting ready to merge their branch into the main repository, then they should consider whether there are any final changes they can make to realign the product’s behavior with the expectations the doc set. If they can’t, then they should share this information with whichever team is responsible for updating documentation as soon as possible so that the team can begin tracking the issue and preparing doc updates for the next release.

Staging environment

The staging environment is the dress rehearsal for releasing software. This is the first environment where everybody with a stake in the documentation should be closely watching the outcome of the tests and scrambling when they fail.

Advantages

1. Pre-release validation: The staging environment mirrors the production environment and is ideal for validating the documentation against the near-final version of the product, ensuring that any discrepancies are caught before public release.
2. Team communication and synergy: This phase fosters collaboration between developers, QA, and technical writers, creating a multidisciplinary review process.
3. Actionable insights: If a test is failing for the docs in the staging environment, it means the test will fail when the software is released. This indicates early to the team that they will need to take some action to fix the problem.

While Docs as Tests works well in each environment, many of its most significant advantages are visible in staging. Developer environments are early enough in the process, but they move fast and most things in them are subject to change. If a test fails in the morning, it may pass in the afternoon once a dev makes their next commit. Production tests are valuable and can alert the team to issues that are already in front of users, but it’s even better to solve problems before users have a chance to see them.

The staging environment is where issues can be detected early enough for teams to have the time to thoughtfully pick and implement the best solution to the discrepancies. Technical writers and other content team members should audit failed tests for deliberate changes to the product and include content and test updates for those tests in the go-to-market plan for the upcoming release.

Production environment

The production environment is the software that users are actually using and is the environment you really want to make sure your docs match. Docs as Tests has you covered for this use case.

Advantages

1. Real-world accuracy: Try as teams might to mitigate them, there are always potential discrepancies in documentation in the live environment, which ensures it reflects the actual user experience, reinforcing user trust and satisfaction.
2. Continuous refinement: The production environment allows for ongoing monitoring and updating of documentation, adapting in real time to product changes and user feedback.
3. Emergency detection: If something goes wrong in production that breaks the accuracy of the documentation, regularly executing doc tests will let you know so your team can spring into action.

Most of the advantages of Docs as Tests for the production environment come from the ability to detect doc discrepancies in the environment where users can find them. The key here is that the tests provide teams with the ability to immediately respond to issues either with patches or doc updates as soon as the problem occurs. With any luck, you never need these tests, but it’s always better to have them than not.

How many environments should you cover?

Ideally, you should cover all of the development environments your product is in. However, things are rarely ideal, so there are a number of ways you can set up coverage depending on how much effort you are able to expend. Let’s go through a few possible implementations.

Decent

• Production: Test once per release (including hotfixes) to verify that all necessary changes were made to the docs. Test failures should trigger issue creation/notifications/investigations to the appropriate teams.

Good

• Staging: Test once per push into the staging environment to verify that all necessary changes were made to the docs for the release. Test failures should trigger notifications/investigations to the appropriate teams. There should be no failures when a feature goes into Production.
• Production: Test once per release (including hotfixes) to verify that all necessary changes were made to the docs. Test failures should trigger issue creation/notifications/investigations to the appropriate teams.

Better

• Staging: Test once per push into the staging environment to verify that all necessary changes were made to the docs for the release. Test failures should trigger notifications/investigations to the appropriate teams. There should be no failures when a feature goes into Production.
• Production:
  • Test once per release (including hotfixes) to verify that all necessary changes were made to the docs. Test failures should trigger issue creation/notifications/investigations to the appropriate teams.
  • Test daily to catch critical, active issues. Test failures should trigger issue creation/notifications/investigations to the appropriate teams.

Best

• Staging: Test once per push into the staging environment to verify that all necessary changes were made to the docs for the release. Test failures should trigger notifications/investigations to the appropriate teams. There should be no failures when a feature goes into Production.
• Production:
  • Test once per release (including hotfixes) to verify that all necessary changes were made to the docs. Test failures should trigger issue creation/notifications/investigations to the appropriate teams.
  • Test daily to catch critical, active issues. Test failures should trigger issue creation/notifications/investigations to the appropriate teams.
• Development: Test in-development features ad-hoc to check feature completeness and doc content accuracy. Test failures should only trigger warnings. There should be no failures when a feature goes into Staging.

The right approach for each environment

Whatever environment you use Docs as Tests in, it will give you more insight into how accurate your docs are compared to the latest version of your product whether that version is being cut for release or just the latest commit during development. What you do with that information varies between environments. During development, it could mean making a couple changes before merging your current feature branch while in staging it may mean planning content updates in advance of the release. In most environments, failed tests should prompt communication between engineering and content teams to make sure everybody knows what needs to happen to make sure users are supported by docs that help them navigate an evolving product.

Today, I’ll walk you through validating both shell commands and scripts, using the commands from the UI validation tutorial as a starting point. We’ll use Doc Detective to validate that the commands work as expected and show how you can run scripts to validate code or commands written in other languages.

(Disclosure: I’m the creator of Doc Detective, but I’ll try to keep this as objective as possible. If you have questions, feel free to reach out in the comments, on social media, or on Discord.)

This is going to get a little meta!

The setup

In the UI validation tutorial, we validated a procedure for searching for American Shorthair kittens on Google, but if you look at all the steps involved, a lot was going on:

1. Install Doc Detective.
2. Save the procedure as a Markdown file.
3. Write a test in JSON.
4. Run the test.
5. Validate the results.

The secret to validating shell commands and scripts is Doc Detective’s runShell action. This action lets you run a shell command or script and validate the results.

We’ll do this a few ways: first, we’ll test individual commands and validate them one at a time, then we’ll create and run a script to perform the whole UI validation procedure in one go, and finally I’ll show you how to validate scripts in whatever language you prefer. Before that though, let’s get Doc Detective ready on your machine.

Install Doc Detective

If you already have Doc Detective installed, skip to the next section.

Before you can get Doc Detective running, you need the prerequisites:

• Node.js 18 or higher. I like using Node Version Manager, or you can use the official installers.
• Git

Once you have those installed, you can clone Doc Detective to your machine and install dependencies:

git clone https://github.com/doc-detective/doc-detective.git
cd doc-detective
npm install

Validate individual commands

Running shell commands and scripts with Doc Detective is straightforward. The command or script runs in your device’s native shell (cmd on Windows and bash on macOS and Linux), and if the command or script returns a non-zero exit code, the test fails.

We can test this out with a single test that has two steps. The first step runs a command that should pass, and the second step runs a command that should fail. Let’s save this test as command-validation.spec.json:

{
  "tests": [
    {
      "id": "single-command-validation",
      "description": "The first step should pass. The second step should fail.",
      "steps": [
        {
          "action": "runShell",
          "command": "echo 'Hello, world!'"
        },
        {
          "action": "runShell",
          "command": "exit 1"
        }
      ]
    }
  ]
}

Now, let’s run the test:

npm run runTests -- -i command-validation.spec.json

Great! We see that the first command passed and the second command failed. This is exactly what we expected because the second command exits with a non-zero exit code. But while these steps test individual commands, you’ll often want to run a script to test multiple commands at once. Let’s look at how to do that.

Validate a bash script

Note: The following commands are for a bash shell. If you’re using Windows, make sure bash is available from cmd, such as through Windows Subsystem for Linux or Git Bash.

Let’s look at each of the steps in the UI validation procedure and identify commands that would complete them:

1. Install Doc Detective:

    git clone https://github.com/doc-detective/doc-detective.git
    cd doc-detective
    npm install
   

2. Save the procedure as a Markdown file:

    echo -e 'To search for American Shorthair kittens,\n\n1. Go to [Google](https://www.google.com).\n\n2. In the search bar, enter \"American Shorthair kittens\", then press Enter.\n\n![Search results](search-results.png)' > kitten-search.md
   

3. Write a test in JSON:

    echo -e '{ "tests": [ { "steps": [ { "action": "goTo", "url": "https://www.google.com" }, { "action": "find", "selector": "[title=Search]", "click": true }, { "action": "typeKeys", "keys": ["American Shorthair kittens", "$ENTER$"] }, { "action": "wait", "duration": 5000 }, { "action": "saveScreenshot", "path": "search-results.png" } ] } ]}' > kitten-search.spec.json
   

4. Run the test:

    npm run runTests -- -i kitten-search.spec.json
   

5. Validate the results, in this case by checking that the screenshot and test results files exist:

    FILE=search-results.png; if [ -f "$FILE" ]; then echo "File $FILE exists."; else false; fi
    for file in testResults-*.json; do [ -e "$file" ] && echo "File $file exists." || false; break; done
   

If you wanted to validate each of these commands individually, you could, but that’s a lot of work. Instead, let’s create a bash script. Save the following script as ui-validation.sh:

# ui-validation.sh
#
# !/bin/bash
git clone https://github.com/doc-detective/doc-detective.git
cd doc-detective
npm install
echo -e 'To search for American Shorthair kittens,\n\n1. Go to [Google](https://www.google.com).\n\n2. In the search bar, enter \"American Shorthair kittens\", then press Enter.\n\n![Search results](search-results.png)' > kitten-search.md
echo -e '{ "tests": [ { "steps": [ { "action": "goTo", "url": "https://www.google.com" }, { "action": "find", "selector": "[title=Search]", "click": true }, { "action": "typeKeys", "keys": ["American Shorthair kittens", "$ENTER$"] }, { "action": "wait", "duration": 5000 }, { "action": "saveScreenshot", "path": "search-results.png" } ] } ]}' > kitten-search.spec.json
npm run runTests -- -i kitten-search.spec.json
FILE=search-results.png; if [ -f "$FILE" ]; then echo "File $FILE exists."; else exit 1; fi
for file in testResults-*.json; do [ -e "$file" ] && echo "File $file exists." || exit 1; break; done

Now, we can validate the whole procedure with a single command. Let’s save this test as ui-validation.spec.json:

{
  "tests": [
    {
      "id": "ui-validation",
      "description": "Validate the UI procedure for searching for American Shorthair kittens on Google.",
      "steps": [
        {
          "action": "runShell",
          "command": "bash ui-validation.sh"
        }
      ]
    }
  ]
}

Now that we have our test and script, let’s run it. Note that while the script is running, we won’t see any output in the terminal. Please be patient.

npm run runTests -- -i ui-validation.spec.json

This script is a little more complex than the others, but it’s a great way to validate a whole procedure in one go.

Validate other kinds of scripts

Script validation isn’t limited to bash. As long as your script can return a non-zero exit code, in can be in any language you like, whether that’s JavaScript, Python, Ruby, or whatever else. Any command you can run from the command line, you can validate with Doc Detective. Need to validate an SDK? Write a script and go for it.

For example, here’s a script that validates only the outputs of our UI procedure. Save this script as validate-kitten-search-outputs.js:

// validate-kitten-search-outputs.js
const fs = require('fs');

const FILE = 'search-results.png';
if (fs.existsSync(FILE)) {
  console.log(`File ${FILE} exists.`);
} else {
  process.exit(1);
}

const files = fs.readdirSync(process.cwd());
const testResults = files.filter(file => file.startsWith('testResults-'));
if (testResults.length > 0) {
  console.log(`File ${testResults[0]} exists.`);
} else {
  process.exit(1);
}

We can test this just as easily as our other script. Save the following test as js-validation.spec.json:

{
  "tests": [
    {
      "id": "js-validation",
      "description": "Validate outputs for searching for American Shorthair kittens on Google.",
      "steps": [
        {
          "action": "runShell",
          "command": "node validate-kitten-search-outputs.js"
        }
      ]
    }
  ]
}

And finally, run the test:

npm run runTests -- -i js-validation.spec.json

Just like the bash script and the individual commands before it, this script exits cleanly. You’re welcome to try deleting the output files and watching it fail too!

One more thing!

If you’re not running Doc Detective as part of a script, you can use runShell to help set up your test environment. For example, you can use runShell to install dependencies, spin up Docker containers, or perform other setup tasks before running your tests. This is a great way to automate your tests and keep them self-contained.

Conclusion

If you want to validate individual commands, to validate scripts in whichever language you require, or to automate your testing environment, Doc Detective is a powerful tool at your disposal. With Doc Detective, you can validate UIs, APIs, and everything in between to make sure your content works the way you and your users expect.

About the tool

Doc Detective is an open-source documentation testing framework. Doc Detective has multiple components to integrate with your workflows as you need it to:

• Doc Detective: A standalone tool to test docs against product UIs and APIs.
• Doc Detective Companion: A browser extension for Chrome and Firefox to help you build tests and find CSS selectors.
• Doc Detective Core: An NPM package that provides Doc Detective’s testing functionality.

Doc Detective is open to contributions, and you can join the chat on Discord. If you like what you see, you can ⭐ star the repositories on GitHub or ❤️ sponsor development via Open Collective or GitHub Sponsors.

Thanks for reading. If you like what you read here, sign up for the newsletter to stay up to date on Docs as Tests discussions, share this article with those you think would appreciate it, or sound off in the comments below.

How are docs supposed to keep up with this rate of change? Technical writers developed two schools of thought that answer these challenges. The first is Docs as Code, an approach to documentation that borrows tools and practices from programming to improve the quality of docs. A more recent school of thought to answer the challenges of rapidly evolving products is Docs as Tests, an approach to using tools to manage the relationship between docs and the product and ensure that docs don’t become stale as the product develops after you publish.

But what is the relationship between Docs as Code and Docs as Tests? They’re similarly named, but how deep do those similarities go? In this post we’ll compare and contrast the ideas and practices associated with these two schools of thought and show that Docs as Tests occupies a unique position as both:

• A natural extension of Docs as Code.
• A standalone set of practices that can be applied outside of Docs as Code workflows.

Docs as Code

Docs as Code (and its more loosely defined sibling Docs like Code) is a philosophy about how to write documentation that borrows heavily from software development.

If you’re a technical writer, you might already know that our job isn’t just about writing technical instructions. When you work in a big organization or on a sizable project, you realize that the complexity grows as the number of instructions increases. Writing technical instructions becomes even more challenging when the documents, content silos, and the domain being covered are too large for any one author to have a full understanding of.

Docs as Code emerged when technical writers realized that another profession faces similar challenges around maintaining enormous, complicated instruction sets: Programmers. Even better, programmers already have practices and tools put together for handling these problems that technical writers can easily retrofit onto their own jobs.

Docs as Code is about studying the day-to-day challenges of managing enormous sets of instructions faced by both technical writers and programmers and trying to figure out how the disciplines can share their learnings to strengthen each other. After you understand those similarities, the tools and practices flow naturally.

Plaintext as the medium of choice

People sometimes treat Docs as Code as a synonym for using plaintext (usually written with a developer-focused text editor) for authoring. Let’s think about the main reasons why Docs as Code favors plaintext markup languages like Markdown or Asciidoc and plaintext structures like Swagger JSON and YAML; it’s not just out of a desire to have technical writers cosplay as developers.

Reason one is all about separating concerns. In your typical word processor, both what the text says and how it looks are controlled through the same interface. This is convenient when you are writing small, one-off documents, but it doesn’t scale. Modern documentation projects are enormous, and updating their visual display to accommodate emerging best practices in web development, new branding guidelines, or even changing tastes is completely impractical to do manually; it must be done programmatically. Plaintext lets the author focus on the authoring, the designer focus on the visual design, and the developer focus on the underlying mechanics of the digital format the docs will be templated into.

Additionally, most plaintext markup languages have open specs that the development community is able to build new tools against. This results in a robust if chaotic ecosystem of options for building publishing toolchains for docs that fit into most dev teams’ skill sets and workflows.

Developer-style version control

Because of its plaintext approach, Docs as Code also works with the gold standard of programming version control tools: Git. There are other version control tools for code but, in practice, Git is the one that sees use.

Working with Git provides more than just the tool to handle version control. If you just write docs without much thought and slap your entire day’s work into a single commit with a description like “Update!” you’re missing out on a lot of the power Docs as Code offers. Programming has a set of best practices for how developers can logically group changes using Git and how to describe them in commits. These practices translate neatly to documentation. When teams follow these practices, the Git history log provides a sort of ‘meta documentation’ about the context of the docs. When this context is explicit:

• New team members have an easier time becoming familiar with the specific design decisions that informed the docs.
• Long-time team members can recall why they made specific decisions about the docs years ago.

Git even has a feature that tells you who to blame for a given section of a document.

Collaboration

Developers write code in teams, and technical writers working on docs can adopt many of the tools that let them work in parallel, share feedback, and organize continuous changes to a highly complex code base. This is where Git, of course, but also Git platforms like Github, Bitbucket, or Gitkraken come into the picture.

Continuous integration and deployment

Treating docs as code also allows the docs to be incorporated into a continuous integration or continuous deployment pipeline where the docs and their publishing toolchain are tested after each committed change to the docs. This helps ensure quality.

In the CI/CD phase of Docs as Code, you will sometimes see the idea of testing the docs mentioned. This practice has a similar name to Docs as Tests but is very different because it’s focused on the testable qualities of the docs such as its syntactic correctness, adherence to style guides, or readability scores.

Docs as Code: The point

Docs as Code focuses on pulling in ideas and tools from software development to improve how docs are written and the specific ‘how’ is focused on how words get written, stored, updated, and tracked.

Docs as Tests

Docs as Test is about using docs as a set of tests for a product. Once the tests are in place, these tests continuously ensure that the documentation correctly describes the functionality of the product. If something changes in the product that makes one of the tests fail, whoever is in charge of the docs can begin identifying the discrepancy and either update the docs to reflect the current state of the product or identify the owners of the ongoing product issue.

Similarly to Docs as Code, Docs as Tests borrows ideas and tools from software engineering and applies them to documentation. However, where Docs as Code is mainly interested in borrowing the tools software developers use to write and manage code and apply them to writing the worst programming language (English) while targeting the most unreliable compiler (the human brain), Docs as Test looks at the ideas behind assertion testing in software development and realizes that they apply to documentation too.

Docs make assertions about the product

From one perspective, documentation is a description of a product. However, in another way, documentation makes falsifiable assertions about a product. When the docs say, “Pressing button A on screen B orders your pizza,” thanks to the power of UI automation, we can test that.

When docs say sending a GET request to the /pizza endpoint returns you a list of all of the objects representing the states of pizza, thanks to the power of API automation, we can test that.

This is the first focus of Docs as Tests: thinking about your documentation as an implicit set of tests that could be built for the documentation.

The Docs are a client

Having tests that break when a product doesn’t perform the way the docs say they do changes the relationship between the docs and the product. It connects documentation to the product in a more tangible way than the “we’ll update it eventually” antipattern that is so easy to fall into when crunch time hits.

Having a slate of tests that alert the team to incongruities between what the docs say and what the product does, especially if the team can detect them ahead of launch, changes the relationship between the docs in a product in a way that somewhat mirrors the relationship between a client and its upstream dependency.

Even if engineering teams don’t see the documents as clients, they might still rethink the value of accurate documents when the tech writer is the one who catches a big bug before the release goes out.

Docs as Tests: The bottom line

Docs as Tests transforms how documentation and products interact, enhancing both their quality. It employs testing tools to identify stale docs as a product evolves and also spots defects from the product. This approach ensures both the documentation and the product stay relevant and error-free, leading to an improved user experience.

Comparing Docs as Tests and Docs as Code

To summarize, Docs as Code applies principles and practices from software development to documentation, emphasizing the use of plain text for writing, version control for tracking changes, and continuous integration and deployment for maintaining quality. This approach treats documentation with the same rigor and discipline as code, allowing for more efficient updates, collaboration, and maintenance.

On the other hand, Docs as Tests takes the relationship between documentation and product functionality a step further by treating documentation as a series of tests for the product. This approach keeps documentation fresh and accurate to the current state of the product by automatically verifying that the features and behaviors described in the documentation match the actual product. If a discrepancy arises, the tool lets teams know so they can fix it.

While Docs as Code provides a robust framework for managing and maintaining documentation, Docs as Tests introduces a dynamic element that directly ties the accuracy of documentation to the product’s functionality. Both approaches offer significant benefits, but they are not mutually exclusive. In fact, integrating Docs as Tests within a Docs as Code framework can create a comprehensive documentation strategy that is both well-organized and consistently accurate, reflecting the current capabilities of the product. This synergy enhances the value of documentation, making it a more reliable and useful resource for users and an integral part of the development process.

With all of this in mind, let’s return to the ideas from the beginning of this post.

Docs as Tests are a natural extension of Docs as Code

Docs as Tests is a natural extension to Docs as Code. A quick survey of both philosophies turns up several overlapping ideas. Both Docs as Code and Docs as Tests

• pull documentation into a paradigm of tools and ideas familiar to developers.
• integrate neatly into developer-centric workflows.
• reduce the number of points where they need to context switch.
• are helpful on teams where developers are involved in the documentation process.

In many ways, Docs as Code contains Docs as Tests within its set of ideas and tools. While Docs as Code started by taking on engineers’ authoring tooling and processes, Docs as Tests continues the trend by mapping engineers’ code testing practices to content written in plain language.

Docs as Tests are a standalone set of practices

One of the beautiful things about Docs as Tests is that you can use it as a framework even if your docs barely resemble code. You could document in Wordpress, a Notion Wiki, an MS Word document, or even a set of instructions handwritten on a piece of paper could be a valid use case with Docs as Tests. Never actually do that last one, but if your life was ever so out of control that you needed to, you totally could.

In a practical sense, however, it would be very difficult to do Docs as Tests for non-Docs as Code workflows right now. This is due to a lack of good tools rather than an incompatibility between the Docs as Tests theory and non-Docs as Code workflows. The tooling available for Docs as Tests is primarily geared towards code-based documentation, but this does not mean that there will never be tools that integrate Docs as Tests into something like Sanity.io or WordPress. Eventually, somebody might come along and build an extension to your CMS of choice that integrates Docs as Tests right into the authoring UI, but that day hasn’t come yet.

Wrapping up

If there’s one key take away from this article, it’s that while Docs as Code and Docs as Test are their own distinct ideas, they complement each other and are a winning combination. The two schools of thought are natural partners at both the philosophical and tooling level.

Docs as Code treats docs as… well, code. This mental model opens up a broad range of ideas and tools in software engineering for use in technical writing. Docs as Tests argues that documentation implies a slate of tests that can be run against a product. Programming as a discipline already has a robust and well-developed set of practices for unit tests that you can immediately apply using any of the many tools available for testing. Since your docs already live in a Git repository as code, you can add code files such as test specs into the same place that your docs are stored and edit them. If you use Doc Detective, a tool specifically built for Docs as Tests, you can even add the tests inline in your Markdown files.

If you’d like to see how to do documents as tests, check out any of the our tutorials.