Run tests in CI/CD
Run your Manufact eval suites from GitHub Actions or any CI provider, fail the job when a suite fails, and report results in the job log, the job summary and the pull request.
You can run your eval suites from CI, so a workflow fails when your MCP server stops passing them. The Manufact Evals GitHub Action starts the suites you pick, waits for the results, and reports them in the job log, the job summary and the pull request. On other CI providers, run the same suites with the mcp-use evals CLI commands.
Evals run against production
The action runs suites against your server's production deployment and has no branch input. Run it after production has deployed: if your deploy runs in the same workflow, put the eval step after it. On a pull request, the results describe production, not the pull request's changes.
Because the action tests production, these triggers fit it best:
| Trigger | When to use it |
|---|---|
push to main | Test production after each release, once the deploy finishes. |
schedule | Catch regressions from model or client changes, even when your code didn't change. |
workflow_dispatch | Run the suites by hand from the Actions tab. |
Before you start
- Create a Manufact account. Sign up at manufact.com.
- Deploy or connect your MCP server. Deploy it from GitHub, or connect a server you host elsewhere.
- Create eval suites. Create them in the dashboard under Testing, or keep them in the repository as YAML files.
- Create an API key in the organization that owns the server. See API keys.
- Store the key as a repository secret. In your GitHub repository, go to Settings → Secrets and variables → Actions and add the key as a secret named
MANUFACT_TOKEN. - Find your server ID. In the examples below, replace
<server-id>with it.
Find your server ID
The server ID is a UUID that identifies your server in Manufact. It is not a secret, so you can put it in the workflow file directly.
In the dashboard: open the server. The server ID is the UUID after /servers/ in the browser's address bar:
https://manufact.com/cloud/<org>/servers/e3b0c442-98fc-4c14-9afb-f4c8996fb924/overview
└─────────── server ID ────────────┘With the CLI: run mcp-use servers list. It prints each server's name, status, and full ID.
Run evals with the GitHub Action
This workflow runs every enabled suite of the server after each push to main, every day at 06:00 UTC, and on demand. The job fails if any suite fails:
name: Evals
on:
push:
branches: [main]
schedule:
- cron: "0 6 * * *" # every day at 06:00 UTC
workflow_dispatch:
concurrency:
group: manufact-evals
cancel-in-progress: true
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: mcp-use/evals@v1
with:
token: ${{ secrets.MANUFACT_TOKEN }}
server: <server-id>The concurrency group cancels a workflow run that is still in progress when a newer one starts, and cancelling the job cancels the eval runs it started. See Costs, timeouts and concurrency.
The action needs a runner with jq. GitHub-hosted runners have it, and the action installs Node.js 22 itself.
Pick suites
By default the action runs every enabled suite of the server. To run some of them, list them in suites, one per line. Each line is a suite ID, a suite name, or the path to a suite file or folder in the repository:
- uses: mcp-use/evals@v1
with:
token: ${{ secrets.MANUFACT_TOKEN }}
server: <server-id>
suites: |
Rome weather
8d0f6b1e-2a4c-4e8b-9f3a-6c1d2e7b5a90- Suite ID. Always matches exactly one suite.
- Suite name. Matches exactly first, then ignoring case. If several suites share the name, the step fails and lists their IDs. Use an ID instead.
- Path to a suite file or folder. Runs the suite named in each file. A path doesn't upload anything: the suite must already exist on Manufact. Paths need
actions/checkoutfirst.
Names can contain spaces and commas, so each line is one suite. A line can't start with -, because the CLI reads it as an option. Use the suite's ID instead.
Keep suites in the repository
You can write suites as YAML files in your repository and push them to Manufact with mcp-use evals push. Each file defines one suite:
name: Rome weather # required; push matches suites by name
flow: # required; at least one user step
- kind: user
content: What's the weather in Rome?
- kind: tool_call # the model must call this tool
name: get_weather
args: { city: { contains: rome } }
rubric: Gives the current temperature in Rome.
threshold: 0.7 # 0 to 1 (default: 0.7)
clients: # default: Inspector with the default model
inspector:
models: [openai/gpt-5]
disallowedTools: [delete_city]
enabled: true # false excludes the suite from runs of every suite- A file is the suite's whole definition. Pushing it over an existing suite resets every field the file leaves out to its default.
mcp-use evals pushcreates or replaces one suite per file, matched byname. A folder contributes its.yaml,.yml, and.jsonfiles, not its subfolders.- Deleting a file does not delete its suite. Delete it with
mcp-use evals rm <suite>. - To move a suite you built in the dashboard into the repository, run
mcp-use evals export <suite> --out evals/<name>.yaml.
Push the files before running them, so the suites on Manufact match the default branch:
name: Evals
on:
push:
branches: [main]
concurrency:
group: manufact-evals
cancel-in-progress: true
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Push eval suites
run: npx --yes @mcp-use/cli evals push evals/
env:
MANUFACT_TOKEN: ${{ secrets.MANUFACT_TOKEN }}
MANUFACT_SERVER: <server-id>
- uses: mcp-use/evals@v1
with:
token: ${{ secrets.MANUFACT_TOKEN }}
server: <server-id>
suites: evals/Push only from your default branch
A push replaces the suite on Manufact for everyone. Pushing from a pull request would change the suites before the change is reviewed.
Run as different accounts
By default the suites sign in with the server's own profile. Set profile to sign in with another profile instead, for example to run the same suites as a free-tier and a paid-tier test user:
jobs:
evals:
runs-on: ubuntu-latest
strategy:
matrix:
profile:
- <free-tier-profile-id>
- <paid-tier-profile-id>
steps:
- uses: mcp-use/evals@v1
with:
token: ${{ secrets.MANUFACT_TOKEN }}
server: <server-id>
profile: ${{ matrix.profile }}- The profile must belong to the server's organization, and be yours or shared with the organization.
- With a profile, every execution signs in, even when the server accepts anonymous calls.
- Native apps, and chatgpt.com or claude.ai when the suite runs the plugin lifecycle, sign in with the suite's test account, so a profile can't apply to them. Manufact refuses the run and the step fails.
- If Manufact doesn't apply the profile, the CLI cancels the run and the step fails, so results never come from the wrong account.
Use the results in later steps
Set fail-on: none to keep the job green and decide what to do with the outputs yourself:
- id: evals
uses: mcp-use/evals@v1
with:
token: ${{ secrets.MANUFACT_TOKEN }}
server: <server-id>
fail-on: none
- if: steps.evals.outputs.passed == 'false'
run: echo "${{ steps.evals.outputs.failed-count }} suites failed"With fail-on: none, the step fails only when the suites can't run at all, for example because the API key is wrong or a suite name doesn't match. Failed suites and timeouts become warnings.
results-file is the path to a JSON file with every run and its executions. Read it with jq to build your own report or notification.
Pull request comment
On pull_request events, the action comments the results on the pull request. Push, schedule, and manual runs don't comment. Their results are in the job log and the job summary.
The comment has a table of suites with their result and score, then a section per suite with a row per client and model showing its score, duration, and tool calls. Suites that didn't pass start expanded. Each failed execution lists why it failed, the judge's assessment, a link to its report in the dashboard, and a collapsed conversation with the user's messages, the tool calls, and the final answer.
- The action keeps one comment per server and
suitesselection, and updates it on later runs. Several eval jobs can share a pull request. - If the suites can't run at all, the comment shows the error instead.
- Commenting needs
pull-requests: write. Without it, the action logs a warning and the eval result is unaffected. - Pull requests from forks get a read-only token, so they get no comment.
- Set
comment: falseto turn comments off.
Remember that the results describe production, not the pull request's changes.
Costs, timeouts and concurrency
- Every suite run is billed. Each suite the action starts is one eval run, billed like a run started from the dashboard. See usage rates. Pick a trigger that matches your budget: pushes to
mainor a schedule cost less than every pull request. - Cancelling the job cancels its runs. When you cancel the job, or GitHub cancels it, the action cancels the eval runs it started that haven't finished. A cancelled run still counts as one eval run once any of its executions has completed.
- Runs keep going after a timeout. When the job reaches
timeout(default 1800 seconds), the runs still going keep running on Manufact and are billed as usual. Open the dashboard to see their results. - Timeouts fail the step. With the default
fail-on: failed, a timeout fails the step. Unfinished runs count infailed-count, andpassedisfalse. - Avoid overlapping runs with a concurrency group. With
cancel-in-progress: true, a new workflow run cancels the one in progress, and with it the eval runs that job started. Leavecancel-in-progressout to queue the new workflow run behind the current one instead.
concurrency:
group: manufact-evals
cancel-in-progress: trueRun evals on any CI provider
The action wraps the mcp-use evals run --wait command of the mcp-use CLI. On another CI provider, run the CLI directly:
npx --yes @mcp-use/cli evals push evals/ # optional: only on your default branch
npx --yes @mcp-use/cli evals run --all --wait \
--json-file evals-results.json \
--markdown-file evals-report.mdSet these environment variables in the CI job:
| Variable | Value |
|---|---|
MANUFACT_TOKEN | Your API key, stored as a CI secret |
MANUFACT_SERVER | Your server ID |
How mcp-use evals run behaves:
- It starts one run per suite against the server's production deployment. Pass suite IDs, names, or paths instead of
--allto run some suites.--allruns every enabled suite. - With
--wait, it polls until every run finishes and exits1unless every run passes. It prints the same live log as the action. --profile <id>signs in with that profile instead of the server's own. See Run as different accounts.--timeout <seconds>sets how long--waitwaits. The default is 1800 seconds. Runs still going at the timeout keep running on Manufact.- Stopping the command while it waits (Ctrl+C, or a cancelled CI job sending
SIGINTorSIGTERM) cancels the runs it started that haven't finished. The command then exits with code130or143. --json-filewrites the results as JSON, and--markdown-filewrites a Markdown report for pull request comments and job summaries. On a timeout, both files still get the runs as they stand.
Inputs
| Input | Default | Description |
|---|---|---|
token | Required | Manufact API key for the server's organization. Store it as a secret. |
server | Required | Server ID. See Find your server ID. |
suites | Every enabled suite | Suites to run, one per line: IDs, names, or paths to suite files or folders. See Pick suites. |
profile | The server's own | ID of a profile of the server's organization to sign in with instead. See Run as different accounts. |
timeout | 1800 | Seconds to wait for the runs. Runs still going at the timeout keep running on Manufact, and the step fails unless fail-on is none. Cancelling the job cancels them instead. |
fail-on | failed | failed fails the job when a suite fails, is cancelled, or times out. none fails it only when the suites can't run at all. |
comment | true | On pull requests, comment the results. Set false to turn it off. |
github-token | github.token | Token used to post the comment. |
Outputs
| Output | Description |
|---|---|
run-ids | Eval run IDs, one per suite, separated by spaces |
passed | true when every suite passed, false otherwise; empty when the suites couldn't run |
passed-count | Number of suites that passed |
failed-count | Number of suites that failed, were cancelled, or were still running at the timeout |
results-file | Path to the JSON results, with every run and its executions |
dashboard-url | Link to the server's eval suites in the dashboard |