Validate README Commands with Innovation Engine
Manny Silva
by Manny Silva
Technical writer by day and engineer by night, Manny Silva is Head of Documentation at Skyflow, the creator of Docs as Tests, and the creator of Doc Detective. He's passionate about intuitive and scalable developer experiences and likes diving into the deep end as the 0th developer.
3 min read


Maintaining accurate and reliable CLI-based documentation is critical, especially when it comes to installation instructions and READMEs.

Azure’s Innovation Engine, introduced at the Linux Foundation Open Source Summit North America 2024, transforms Markdown into executable scripts, enabling interactive tutorials, automated tests, response validation, and execution of CLI commands. While Innovation Engine is still in the early stages of development, it shows promise for improving the quality and accuracy of CLI-based docs.

Here, we’ll cover how to use Innovation Engine to test all the bash commands in code blocks within a tutorial written in Markdown. We’ll use a simple tutorial that interacts with the ReqRes API, a fake online REST API designed for testing and prototyping. One item to note, though: Innovation Engine is bash-only at the moment, so you’ll need a macOS or Linux environment (Windows Subsystem for Linux (WSL) on Windows works great) to follow along.


This tutorial is slightly more advanced than other recent ones. Let’s make sure you’re ready to go. You need

  • A bash shell.
  • Go installed on your machine.

Set up Innovation Engine

To install and build Innovation Engine, you need to download or build the ie binary.

On Linux, you can download the binary and make it executable with the following commands:

wget -q -O ie$VERSION/ie
chmod +x ie

On macOS (or if you like to go the hard road on Linux), you need to clone the repo and run the build command:

git clone;
cd InnovationEngine;
make build-ie;
cd ./bin

Sample document and formatting

Here’s a simple tutorial in Markdown that uses ReqRes, a testing API that uses fake data. The tutorial focuses on two common API requests: fetching a list of users and creating a new user.

Each command is in a bash code block, and the expected output is in a JSON code block below it. By default, Innovation Engine runs commands in bash code blocks to make sure they run as expected. For example:

    curl -X POST \
      -H "Content-Type: application/json" \
      -d '{
        "name": "Jane Doe",
        "job": "Developer"

To validate responses, you need to add an HTML-style comment before the immediately following code block and include the expected_similarity key and a value between 0 and 1. This value represents the similarity between the expected output and the actual output, allowing for fuzzy matching.

    {"name":"Jane Doe","job":"Developer","id":"557","createdAt":"2024-07-04T21:58:42.684Z"}

Note: Response code blocks can’t have language tags, so you can’t use json or bash here. If there’s a language tag, Innovation Engine won’t consider it a response block.

Testing the commands

Using Innovation Engine, we can test the commands in the code blocks to make sure they run correctly.

  1. Download into the same directory at the ie binary.
  2. Run the following command to test the commands in the tutorial:

     ./ie test

    This command reads the Markdown file, executes the curl commands, and validates the output.

    The output should look similar to this:

     $ ./ie test
     bash:$ curl
     {"page":1,"per_page":6,"total":12,"total_pages":2,"data":[{"id":1,"email":"[email protected]","first_name":"George","last_name":"Bluth","avatar":""},{"id":2,"email":"[email protected]","first_name":"Janet","last_name":"Weaver","avatar":""},{"id":3,"email":"[email protected]","first_name":"Emma","last_name":"Wong","avatar":""},{"id":4,"email":"[email protected]","first_name":"Eve","last_name":"Holt","avatar":""},{"id":5,"email":"[email protected]","first_name":"Charles","last_name":"Morris","avatar":""},{"id":6,"email":"[email protected]","first_name":"Tracey","last_name":"Ramos","avatar":""}],"support":{"url":"","text":"To keep ReqRes free, contributions towards server costs are appreciated!"}}
     bash:$ curl -X POST \
       -H "Content-Type: application/json" \
       -d '{
         "name": "Jane Doe",
         "job": "Developer"
     {"name":"Jane Doe","job":"Developer","id":"334","createdAt":"2024-07-04T22:17:31.936Z"}

    The output shows the commands being executed and the expected output. If the actual output matches the expected output within the similarity threshold, the test passes. If the output isn’t similar enough to the expected output, the test fails.

Wrapping up

By validating your documentation with Innovation Engine, you can make sure that your code examples are always accurate and functional. While Innovation Engine is limited to testing CLI docs, it can be integrated into your CI/CD pipeline to automate your testing and make sure that your most important CLI-based docs, like installation instructions, are always up-to-date and accurate.

About the tool

Azure’s Innovation Engine is a CLI tool that transforms markdown documentation into executable scripts. This allows for interactive tutorials, automated tests, and execution of CLI commands, making sure your documentation remains accurate and up-to-date. For more information, see the Innovation Engine GitHub repository.

Thanks for reading. If you like what you read here, sign up for the newsletter to stay up to date on Docs as Tests discussions, share this article with those you think would appreciate it, or sound off in the comments below.