Run tests in CI/CD

Run your Manufact eval suites from GitHub Actions or any CI provider, fail the job when a suite fails, and report results in the job log, the job summary and the pull request.

You can run your eval suites from CI, so a workflow fails when your MCP server stops passing them. The Manufact Evals GitHub Action starts the suites you pick, waits for the results, and reports them in the job log, the job summary and the pull request. On other CI providers, run the same suites with the mcp-use evals CLI commands.

Evals run against production

The action runs suites against your server's production deployment and has no branch input. Run it after production has deployed: if your deploy runs in the same workflow, put the eval step after it. On a pull request, the results describe production, not the pull request's changes.

Because the action tests production, these triggers fit it best:

TriggerWhen to use it
push to mainTest production after each release, once the deploy finishes.
scheduleCatch regressions from model or client changes, even when your code didn't change.
workflow_dispatchRun the suites by hand from the Actions tab.

Before you start

  1. Create a Manufact account. Sign up at manufact.com.
  2. Deploy or connect your MCP server. Deploy it from GitHub, or connect a server you host elsewhere.
  3. Create eval suites. Create them in the dashboard under Testing, or keep them in the repository as YAML files.
  4. Create an API key in the organization that owns the server. See API keys.
  5. Store the key as a repository secret. In your GitHub repository, go to Settings → Secrets and variables → Actions and add the key as a secret named MANUFACT_TOKEN.
  6. Find your server ID. In the examples below, replace <server-id> with it.

Find your server ID

The server ID is a UUID that identifies your server in Manufact. It is not a secret, so you can put it in the workflow file directly.

In the dashboard: open the server. The server ID is the UUID after /servers/ in the browser's address bar:

https://manufact.com/cloud/<org>/servers/e3b0c442-98fc-4c14-9afb-f4c8996fb924/overview
                                         └─────────── server ID ────────────┘

With the CLI: run mcp-use servers list. It prints each server's name, status, and full ID.

Run evals with the GitHub Action

This workflow runs every enabled suite of the server after each push to main, every day at 06:00 UTC, and on demand. The job fails if any suite fails:

.github/workflows/evals.yml
name: Evals

on:
  push:
    branches: [main]
  schedule:
    - cron: "0 6 * * *" # every day at 06:00 UTC
  workflow_dispatch:

concurrency:
  group: manufact-evals
  cancel-in-progress: true

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: mcp-use/evals@v1
        with:
          token: ${{ secrets.MANUFACT_TOKEN }}
          server: <server-id>

The concurrency group cancels a workflow run that is still in progress when a newer one starts, and cancelling the job cancels the eval runs it started. See Costs, timeouts and concurrency.

The action needs a runner with jq. GitHub-hosted runners have it, and the action installs Node.js 22 itself.

Pick suites

By default the action runs every enabled suite of the server. To run some of them, list them in suites, one per line. Each line is a suite ID, a suite name, or the path to a suite file or folder in the repository:

      - uses: mcp-use/evals@v1
        with:
          token: ${{ secrets.MANUFACT_TOKEN }}
          server: <server-id>
          suites: |
            Rome weather
            8d0f6b1e-2a4c-4e8b-9f3a-6c1d2e7b5a90
  • Suite ID. Always matches exactly one suite.
  • Suite name. Matches exactly first, then ignoring case. If several suites share the name, the step fails and lists their IDs. Use an ID instead.
  • Path to a suite file or folder. Runs the suite named in each file. A path doesn't upload anything: the suite must already exist on Manufact. Paths need actions/checkout first.

Names can contain spaces and commas, so each line is one suite. A line can't start with -, because the CLI reads it as an option. Use the suite's ID instead.

Keep suites in the repository

You can write suites as YAML files in your repository and push them to Manufact with mcp-use evals push. Each file defines one suite:

evals/weather.yaml
name: Rome weather           # required; push matches suites by name
flow:                        # required; at least one user step
  - kind: user
    content: What's the weather in Rome?
  - kind: tool_call          # the model must call this tool
    name: get_weather
    args: { city: { contains: rome } }
rubric: Gives the current temperature in Rome.
threshold: 0.7               # 0 to 1 (default: 0.7)
clients:                     # default: Inspector with the default model
  inspector:
    models: [openai/gpt-5]
disallowedTools: [delete_city]
enabled: true                # false excludes the suite from runs of every suite
  • A file is the suite's whole definition. Pushing it over an existing suite resets every field the file leaves out to its default.
  • mcp-use evals push creates or replaces one suite per file, matched by name. A folder contributes its .yaml, .yml, and .json files, not its subfolders.
  • Deleting a file does not delete its suite. Delete it with mcp-use evals rm <suite>.
  • To move a suite you built in the dashboard into the repository, run mcp-use evals export <suite> --out evals/<name>.yaml.

Push the files before running them, so the suites on Manufact match the default branch:

.github/workflows/evals.yml
name: Evals

on:
  push:
    branches: [main]

concurrency:
  group: manufact-evals
  cancel-in-progress: true

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Push eval suites
        run: npx --yes @mcp-use/cli evals push evals/
        env:
          MANUFACT_TOKEN: ${{ secrets.MANUFACT_TOKEN }}
          MANUFACT_SERVER: <server-id>
      - uses: mcp-use/evals@v1
        with:
          token: ${{ secrets.MANUFACT_TOKEN }}
          server: <server-id>
          suites: evals/

Push only from your default branch

A push replaces the suite on Manufact for everyone. Pushing from a pull request would change the suites before the change is reviewed.

Run as different accounts

By default the suites sign in with the server's own profile. Set profile to sign in with another profile instead, for example to run the same suites as a free-tier and a paid-tier test user:

.github/workflows/evals.yml
jobs:
  evals:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        profile:
          - <free-tier-profile-id>
          - <paid-tier-profile-id>
    steps:
      - uses: mcp-use/evals@v1
        with:
          token: ${{ secrets.MANUFACT_TOKEN }}
          server: <server-id>
          profile: ${{ matrix.profile }}
  • The profile must belong to the server's organization, and be yours or shared with the organization.
  • With a profile, every execution signs in, even when the server accepts anonymous calls.
  • Native apps, and chatgpt.com or claude.ai when the suite runs the plugin lifecycle, sign in with the suite's test account, so a profile can't apply to them. Manufact refuses the run and the step fails.
  • If Manufact doesn't apply the profile, the CLI cancels the run and the step fails, so results never come from the wrong account.

Use the results in later steps

Set fail-on: none to keep the job green and decide what to do with the outputs yourself:

      - id: evals
        uses: mcp-use/evals@v1
        with:
          token: ${{ secrets.MANUFACT_TOKEN }}
          server: <server-id>
          fail-on: none
      - if: steps.evals.outputs.passed == 'false'
        run: echo "${{ steps.evals.outputs.failed-count }} suites failed"

With fail-on: none, the step fails only when the suites can't run at all, for example because the API key is wrong or a suite name doesn't match. Failed suites and timeouts become warnings.

results-file is the path to a JSON file with every run and its executions. Read it with jq to build your own report or notification.

Pull request comment

On pull_request events, the action comments the results on the pull request. Push, schedule, and manual runs don't comment. Their results are in the job log and the job summary.

The comment has a table of suites with their result and score, then a section per suite with a row per client and model showing its score, duration, and tool calls. Suites that didn't pass start expanded. Each failed execution lists why it failed, the judge's assessment, a link to its report in the dashboard, and a collapsed conversation with the user's messages, the tool calls, and the final answer.

  • The action keeps one comment per server and suites selection, and updates it on later runs. Several eval jobs can share a pull request.
  • If the suites can't run at all, the comment shows the error instead.
  • Commenting needs pull-requests: write. Without it, the action logs a warning and the eval result is unaffected.
  • Pull requests from forks get a read-only token, so they get no comment.
  • Set comment: false to turn comments off.

Remember that the results describe production, not the pull request's changes.

Costs, timeouts and concurrency

  • Every suite run is billed. Each suite the action starts is one eval run, billed like a run started from the dashboard. See usage rates. Pick a trigger that matches your budget: pushes to main or a schedule cost less than every pull request.
  • Cancelling the job cancels its runs. When you cancel the job, or GitHub cancels it, the action cancels the eval runs it started that haven't finished. A cancelled run still counts as one eval run once any of its executions has completed.
  • Runs keep going after a timeout. When the job reaches timeout (default 1800 seconds), the runs still going keep running on Manufact and are billed as usual. Open the dashboard to see their results.
  • Timeouts fail the step. With the default fail-on: failed, a timeout fails the step. Unfinished runs count in failed-count, and passed is false.
  • Avoid overlapping runs with a concurrency group. With cancel-in-progress: true, a new workflow run cancels the one in progress, and with it the eval runs that job started. Leave cancel-in-progress out to queue the new workflow run behind the current one instead.
concurrency:
  group: manufact-evals
  cancel-in-progress: true

Run evals on any CI provider

The action wraps the mcp-use evals run --wait command of the mcp-use CLI. On another CI provider, run the CLI directly:

npx --yes @mcp-use/cli evals push evals/   # optional: only on your default branch
npx --yes @mcp-use/cli evals run --all --wait \
  --json-file evals-results.json \
  --markdown-file evals-report.md

Set these environment variables in the CI job:

VariableValue
MANUFACT_TOKENYour API key, stored as a CI secret
MANUFACT_SERVERYour server ID

How mcp-use evals run behaves:

  • It starts one run per suite against the server's production deployment. Pass suite IDs, names, or paths instead of --all to run some suites. --all runs every enabled suite.
  • With --wait, it polls until every run finishes and exits 1 unless every run passes. It prints the same live log as the action.
  • --profile <id> signs in with that profile instead of the server's own. See Run as different accounts.
  • --timeout <seconds> sets how long --wait waits. The default is 1800 seconds. Runs still going at the timeout keep running on Manufact.
  • Stopping the command while it waits (Ctrl+C, or a cancelled CI job sending SIGINT or SIGTERM) cancels the runs it started that haven't finished. The command then exits with code 130 or 143.
  • --json-file writes the results as JSON, and --markdown-file writes a Markdown report for pull request comments and job summaries. On a timeout, both files still get the runs as they stand.

Inputs

InputDefaultDescription
tokenRequiredManufact API key for the server's organization. Store it as a secret.
serverRequiredServer ID. See Find your server ID.
suitesEvery enabled suiteSuites to run, one per line: IDs, names, or paths to suite files or folders. See Pick suites.
profileThe server's ownID of a profile of the server's organization to sign in with instead. See Run as different accounts.
timeout1800Seconds to wait for the runs. Runs still going at the timeout keep running on Manufact, and the step fails unless fail-on is none. Cancelling the job cancels them instead.
fail-onfailedfailed fails the job when a suite fails, is cancelled, or times out. none fails it only when the suites can't run at all.
commenttrueOn pull requests, comment the results. Set false to turn it off.
github-tokengithub.tokenToken used to post the comment.

Outputs

OutputDescription
run-idsEval run IDs, one per suite, separated by spaces
passedtrue when every suite passed, false otherwise; empty when the suites couldn't run
passed-countNumber of suites that passed
failed-countNumber of suites that failed, were cancelled, or were still running at the timeout
results-filePath to the JSON results, with every run and its executions
dashboard-urlLink to the server's eval suites in the dashboard

On this page