> ## Documentation Index > Fetch the complete documentation index at: https://mcpjam-mintlify-docs-update-pr-4053-1786992934526.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Create an eval run (async) > Creates a suite run from an existing `suiteId` (rerun) and/or inline `tests`, then **detaches execution and responds `202` immediately** with the `runId`. Validation and quota errors surface on this request; poll `GET /eval-runs/{runId}` for progress. The run appears live in the hosted UI Runs tab, tagged `source: "api"`. A bare `suiteId` with no inline tests reruns the suite as configured. Per-organization concurrency is capped (default 2 concurrent runs); exceeding it returns `429` with `details.reason: "CONCURRENT_RUN_LIMIT"`. For a suite with attached project environments, pass `environmentId` to choose which one the run uses; the `202` echoes the resolved `environment` triple, and `GET /eval-runs/{runId}` reports the same triple for the life of the run. ## OpenAPI ````yaml /reference/openapi.json post /projects/{projectId}/eval-runs openapi: 3.1.0 info: title: MCPJam API version: 1.0.0-preview description: >- Programmatic access to MCP servers saved in your MCPJam projects — live diagnostics (validate, inspect, export) and operations: call tools, render prompts, run eval suites asynchronously and poll their results, and import OAuth tokens. **The API is in preview**: the surface may change while we finish the design. Error `code` values are stable; error `message` strings are not. Write clients that ignore unknown response fields. contact: name: MCPJam url: https://github.com/MCPJam/inspector/issues servers: - url: https://app.mcpjam.com/api/v1 description: Hosted MCPJam security: - bearerAuth: [] tags: - name: Hosts description: >- Project hosts: named model + capability profiles you run chats and eval suites against. - name: Environments description: >- Project environments: named, live-editable execution bundles (one host, an optional standalone server group, optionally pinned skills and plugin versions) that eval suites and journeys run against. Distinct from Sandbox images, which are Computer base images. Reads require project membership; every write requires project admin. - name: Plugins description: >- Agent Plugins imported into a project — read-only inventory and version detail. - name: Sandbox images description: >- Custom Computer images: a digest-pinned Dockerfile built into an immutable image your project's computers boot from. - name: Server diagnostics description: Connect-level health checks against a saved MCP server. - name: Primitives description: 'The server''s MCP primitives: tools, prompts, and resources.' - name: Export description: Full-server snapshots for diffing and CI. - name: Execution description: 'Run the server''s primitives: call tools, render prompts.' - name: Eval runs description: >- Asynchronous eval suite runs: create with 202, poll status, iterations, and traces. - name: Conformance runs description: >- Ingest MCP spec-conformance results from the SDK/CLI into project-owned history. Distinct from Eval runs (authored LLM cases) and from directory readiness. - name: Server connections description: >- Connect an MCP server URL to a project, authorizing in a browser when the server requires it. - name: OAuth description: 'Bring-your-own OAuth: import externally obtained tokens for a server.' - name: Scenarios description: >- Read-only access to the scenarios published from a project: listing, settings, attached servers, and share links. - name: Catalog description: >- Discover the resources the other routes operate on: your account, projects, servers, eval suites, and chat sessions. - name: Tunnels description: >- Relay tunnels that expose local MCP servers through a public URL, registered as first-class project servers (the `mcpjam cloud tunnel` CLI flow). - name: Agent description: >- Headless agent turns over the public API: send a message history, the server runs one assistant turn with project-scoped workspace tools (eval reads + suite creation) on a pinned hosted model, and returns the reply plus created-resource references. - name: Swarms description: >- Personas, journeys and swarm containers — the authoring half of Swarms — plus the model-backed generation that drafts them. - name: Swarm runs description: >- Launching journeys and reading what they produced. Launching SPENDS — see the per-operation notes. - name: Swarm insights description: >- What a swarm run revealed. The scorecard and findings are deterministic and free; requesting wave insights runs models and draws on your shared daily ledger. - name: User testing description: >- Publishing an environment for real visitors, and controlling who can reach it. Several of these NARROW access and take effect immediately. - name: Directory readiness description: >- Grade a saved server against a publisher's listing requirements: Anthropic's connector directory or OpenAI's plugin directory. Reported as lane status and coverage, never as a numeric score, and excluded from `pooledConformanceScore`. Deterministic grading is free; model-backed experience observations are an explicit opt-in that consumes MCPJam credits and can never decide a verdict. - name: Registry description: >- Search the scraped MCP directories (Claude, ChatGPT, and any future source), list curated/org registry cards, and install them into a project. Install writes a `servers` row and provenance — it does not open a live session. There is no catalog-uninstall route: delete the project server instead. Directory reads require a bearer (including minted guest tokens) but do not materialize a user. Card/connection reads and all writes are authed-non-guest. paths: /projects/{projectId}/eval-runs: post: tags: - Eval runs summary: Create an eval run (async) description: >- Creates a suite run from an existing `suiteId` (rerun) and/or inline `tests`, then **detaches execution and responds `202` immediately** with the `runId`. Validation and quota errors surface on this request; poll `GET /eval-runs/{runId}` for progress. The run appears live in the hosted UI Runs tab, tagged `source: "api"`. A bare `suiteId` with no inline tests reruns the suite as configured. Per-organization concurrency is capped (default 2 concurrent runs); exceeding it returns `429` with `details.reason: "CONCURRENT_RUN_LIMIT"`. For a suite with attached project environments, pass `environmentId` to choose which one the run uses; the `202` echoes the resolved `environment` triple, and `GET /eval-runs/{runId}` reports the same triple for the life of the run. operationId: createEvalRun parameters: - $ref: '#/components/parameters/projectId' requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/EvalRunCreateRequest' example: suiteName: smoke serverIds: - srv_abc123 tests: - title: echo works runs: 1 model: anthropic/claude-haiku-4.5 provider: anthropic steps: - id: s1 kind: prompt prompt: Use the echo tool to say hi - id: s2 kind: assert assertion: type: toolCalledWith toolName: echo args: args: text: hi responses: '202': description: Run created; execution continues in the background. content: application/json: schema: $ref: '#/components/schemas/EvalRunCreated' example: runId: run_abc123 suiteId: suite_def456 status: running caseUpsert: committed: - name: echo works failed: [] environment: null '400': $ref: '#/components/responses/ValidationError' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/InternalError' '502': $ref: '#/components/responses/ServerUnreachable' '504': $ref: '#/components/responses/Timeout' components: parameters: projectId: name: projectId in: path required: true description: ID of the hosted project that contains the server. schema: type: string schemas: EvalRunCreateRequest: type: object description: >- Two valid shapes: `suiteId` (rerun an existing suite, optionally upserting inline `tests` into it) or `suiteName` + `tests` + `serverIds` (create a new suite and run it). Inline `tests` alone — without a `suiteId` or a `suiteName` — are rejected with `VALIDATION_ERROR`. `environmentId` requires `suiteId`: an environment is launchable only through a suite that has it attached (`environmentIds`, set via `PATCH /eval-suites/{suiteId}`), so an environment run on a not-yet-created suite could never be satisfied. `environmentId` and `serverIds` are mutually exclusive. anyOf: - required: - suiteId - required: - suiteName - tests - serverIds properties: tests: minItems: 1 properties: suiteId: type: string description: >- Existing suite to rerun. A bare `suiteId` with no `tests` reruns the suite exactly as configured. suiteName: type: string description: >- Name for a new suite. Required (non-empty) when no `suiteId` is given. suiteDescription: type: string tests: type: array maxItems: 100 description: Inline test cases to upsert into the suite before running. items: $ref: '#/components/schemas/EvalTestCase' serverIds: type: array minItems: 1 description: >- Servers (by ID) the run connects to. Required when creating a new suite; optional on reruns — when omitted, the run connects the suite's saved server selection (the set its snapshot references). A rerun of a suite with no saved selection is rejected with `VALIDATION_ERROR` (`details.reason: "NO_SAVED_SERVER_SELECTION"`). Rejected outright for a suite with attached environments (`details.reason: "ENVIRONMENT_SERVERS_NOT_OVERRIDABLE"`): the environment supplies a closed set that a server override cannot change, so accepting one would connect a different set than the run is stamped with. items: type: string serverNames: type: array description: Optional display names, parallel to `serverIds`. items: type: string suiteRerun: type: boolean description: >- When true, skip per-case upsert and rerun the persisted suite. A bare `suiteId` with no `tests` is always treated as a rerun. modelApiKeys: type: object description: >- Optional per-provider model API keys (e.g. `{ "anthropic": "sk-ant-…" }`). Falls back to your organization's configured providers when omitted. additionalProperties: type: string notes: type: string passCriteria: type: object properties: minimumPassRate: type: number iterationOverride: type: integer minimum: 1 maximum: 10 description: Override the per-case `runs` count for this run only. environmentId: type: string description: >- Run against one of the suite's attached project environments. Requires `suiteId`, and must be a member of that suite's `environmentIds` — otherwise `400` with `details.reason: "ENVIRONMENT_NOT_ATTACHED"`, raised before any case is authored or any server connected. Omission is meaningful: a suite with no attached environments runs legacy; a suite with exactly ONE attached environment runs against it automatically (the response's `environment` says which); a suite with several returns `400` with `details.reason: "ENVIRONMENT_REQUIRED"`, naming the candidates. The environment supplies the closed server set (so `serverIds` is not required, and is rejected alongside it), and the run is pinned to the revision resolved at launch — if the environment changes in between, the run is rejected with `409` rather than executing against a different configuration. ephemeralEnvironment: type: boolean description: >- When true, `environmentId` may be a project-scoped, non-archived environment that is NOT attached to the suite. The launch never mutates the suite. Absent / false keeps the membership check. Probe `GET /environments/capabilities` (`ephemeralEnvironmentLaunch`) before sending — older servers reject the unknown field. namedHostId: type: string description: >- Run against ONE host attached to the suite. The platform snapshots that host's current config onto the run and derives the run's server set from it, so a host launch needs no `serverIds`. Without this, a suite with host attachments runs under the suite's own default host config — the run executes, but the result is attributed to the wrong host. To run SEVERAL attached hosts, use `POST /eval-run-groups` rather than N calls here: it is the surface that bounds the fan-out and meters it as one launch. caseIds: type: array minItems: 1 description: >- Narrow the run to these suite cases. The persisted suite is untouched — this filters the run's snapshot only. Every id must belong to the suite; none matching returns `404`. items: type: string matchOptionsOverride: description: >- One-off tool-call match options for this run only, layered over suite defaults and per-case overrides. Does NOT mutate the suite or its cases. Accepts EITHER the public vocabulary (`toolCallOrder: any|in-order|exact`, `extraToolCalls`, `arguments`) or the internal one (`toolCallOrder: ignore|superset|strict`, `maxExtraToolCalls`, `argumentMatching`). The two are disjoint, so a body can only be one of them; public bodies are normalized server-side. type: object properties: toolCallOrder: type: string description: >- Public: `any` (order ignored) | `in-order` (expected calls appear in order, extras allowed) | `exact` (exact sequence). Internal equivalents: `ignore` | `superset` | `strict`. enum: - any - in-order - exact - ignore - superset - strict extraToolCalls: description: >- Public form: `"unlimited"`, or a maximum count of unexpected extra tool calls. oneOf: - type: string enum: - unlimited - type: integer minimum: 0 arguments: type: string description: Public form of argument-comparison strictness. enum: - ignore - partial - exact maxExtraToolCalls: description: 'Internal form of `extraToolCalls`: `null` is unlimited.' type: - integer - 'null' minimum: 0 argumentMatching: type: string description: Internal form of `arguments`. enum: - ignore - partial - exact skillsOverride: type: string enum: - exclude description: >- The "without skills" arm of an A/B comparison: the run pins NO skills from any channel and is marked `skillsExcluded`, so the arm is labelled rather than merely empty. Scoped to skill DELIVERY — a pinned plugin's MCP servers stay connected, because which servers an arm connects is the one variable a skills A/B has to hold fixed. refreshSnapshot: type: boolean description: >- **PERSISTS A SUITE MUTATION.** Re-derives and stores the suite's host-config snapshot from this request's server list, so future runs of the suite use it too. Without it a rerun leaves the snapshot frozen, which is what stops newly connected servers from silently contaminating an existing suite. Single-target launches only — it is not accepted on `POST /eval-run-groups`, where last-writer-wins on a frozen snapshot is never what a fan-out means. runGroupId: type: string description: >- A LABEL that groups sibling run rows for display. It has NO quota or launch semantics here: N calls carrying one id are still N independent launches, each metered separately. Grouped-launch behaviour lives on `POST /eval-run-groups`, which mints the id itself. Echoed back on the `202`. idempotencyKey: type: string maxLength: 256 description: >- Write-idempotency key. A repeat call with the same key (same actor and suite) returns the EXISTING run instead of creating and billing a second one. The `Idempotency-Key` header carries the same value and WINS over this field — it is the transport-level channel unattended clients control, whereas a body key could be shaped by model output. sourceHash: type: string pattern: ^[a-f0-9]{64}$ description: >- SHA-256 hex of the suite-file bytes that launched this run. Lowercase, 64 characters. Set by `eval run --file`; a UI or API launch that did not come from a file omits it. additionalProperties: false EvalRunCreated: type: object required: - runId - suiteId - status - caseUpsert properties: runId: type: string suiteId: type: string status: type: string description: >- The run's status. `running` on a fresh launch; on a replay (see `deduped`), the existing run's own status, which may already be terminal. deduped: type: boolean description: >- Present and true when this request REPLAYED an existing run instead of starting one (an idempotency-key hit, or the short keyless dedupe window). A replayed run is not executed again, so no further credits are spent; read `status` for what that run actually is. Absent on a fresh launch. runGroupId: type: string description: >- Echo of the request's `runGroupId`, when one was sent. A LABEL only — it groups sibling rows for display and carries no quota or launch semantics. servers: type: array description: >- The servers the run connects to — explicit or derived from the suite's saved selection. `name` is present when known (always, on the derived path). items: type: object required: - id properties: id: type: string name: type: string caseUpsert: type: object description: >- Per-case upsert outcomes for inline `tests`. Partial failures don't abort the run. properties: committed: type: array items: type: object properties: id: type: string name: type: string failed: type: array items: type: object properties: id: type: string name: type: string error: type: string environment: description: >- The environment revision this run is pinned to. `null` on a legacy run that recorded none — always present, so a caller never has to distinguish absent from unpinned. oneOf: - $ref: '#/components/schemas/EvalRunEnvironment' - type: 'null' EvalTestCase: type: object description: >- Inline eval test case. The case body is an ordered `steps` array (the unified test model). required: - title - runs - model - provider - steps properties: title: type: string steps: type: array minItems: 1 description: >- Ordered test steps. The first `prompt` step is the case query; `toolCalledWith` asserts are the expected tool calls; a single model-free `toolCall` step is a render-check. items: $ref: '#/components/schemas/EvalTestStep' runs: type: integer minimum: 1 maximum: 10 description: Iterations to execute for this case. model: type: string description: >- Model ID. Hosted-catalog ids use `provider/name` form (e.g. `anthropic/claude-haiku-4.5`) and run on org credits; provider-native ids require a matching `modelApiKeys` entry (BYOK). Unknown models are rejected with VALIDATION_ERROR at create time. provider: type: string description: Model provider, e.g. `anthropic`, `openai`. isNegativeTest: type: boolean description: When `true`, the case passes if NO tools are called. expectedOutput: type: string advancedConfig: type: object description: Optional `system`, `temperature`, `toolChoice` overrides. additionalProperties: true additionalProperties: true EvalRunEnvironment: type: - object - 'null' description: >- The project environment a run is pinned to, at the revision resolved when it launched. `null` for a legacy run that used the suite's saved server selection. required: - id properties: id: type: string name: type: - string - 'null' revision: type: - integer - 'null' description: The environment revision the run executed against. Error: type: object required: - code - message properties: code: type: string description: >- Stable, machine-readable error code. New codes may be added over time; treat unknown codes as non-retryable failures unless the HTTP status says otherwise. enum: - UNAUTHORIZED - FORBIDDEN - NOT_FOUND - CONFLICT - VALIDATION_ERROR - RATE_LIMITED - FEATURE_NOT_SUPPORTED - SERVER_UNREACHABLE - TIMEOUT - OAUTH_REQUIRED - INTERNAL_ERROR message: type: string description: >- Human-readable description. May change between releases — don't match on it. details: type: object description: Optional, unstructured context bag. additionalProperties: true EvalTestStep: type: object description: >- One authored test step (the unified test model). `kind` discriminates: `prompt` is a user message (model turn); `toolCall` is a deterministic, model-free tool call; `interact` is one pure widget action; `assert` is an assertion (a `Predicate` like `toolCalledWith` / `widgetRendered`, or a DOM `WidgetAssertion`). required: - id - kind properties: id: type: string minLength: 1 kind: type: string enum: - prompt - toolCall - interact - assert prompt: type: string description: 'User message (`kind: prompt`).' serverName: type: string description: 'Server that owns the tool (`kind: toolCall`).' toolName: type: string description: 'Tool name (`kind: toolCall` / `interact`).' arguments: type: object description: 'Tool-call arguments (`kind: toolCall`).' additionalProperties: true action: type: object description: 'Widget action (`kind: interact`).' additionalProperties: true assertion: type: object description: 'Predicate or widget assertion (`kind: assert`).' additionalProperties: true additionalProperties: true responses: ValidationError: description: Malformed body or parameters. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: VALIDATION_ERROR message: Invalid JSON body Unauthorized: description: >- Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key. content: application/json: schema: $ref: '#/components/schemas/Error' examples: badKey: summary: Invalid or revoked key value: code: UNAUTHORIZED message: Invalid API key oauthRequired: summary: Target server needs an OAuth grant value: code: OAUTH_REQUIRED message: Server requires OAuth authorization Forbidden: description: Key is valid but not allowed to do this. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: FORBIDDEN message: You do not have access to this project NotFound: description: Unknown project, server, or resource. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: NOT_FOUND message: Server not found RateLimited: description: >- Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter. headers: Retry-After: description: Seconds to wait before retrying. schema: type: integer content: application/json: schema: $ref: '#/components/schemas/Error' example: code: RATE_LIMITED message: API key rate limit exceeded. Slow down and retry. InternalError: description: Something failed on MCPJam's side. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: INTERNAL_ERROR message: Unexpected internal error ServerUnreachable: description: Could not connect to the target MCP server. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: SERVER_UNREACHABLE message: Failed to connect to server Timeout: description: The target MCP server connected but didn't respond in time. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: TIMEOUT message: Request to server timed out securitySchemes: bearerAuth: type: http scheme: bearer description: >- MCPJam API key (`sk_…`). Create one at [Settings → API keys](https://app.mcpjam.com/settings/api-keys). Guest sessions cannot use the API, and API keys cannot manage other API keys. ````