> ## Documentation Index > Fetch the complete documentation index at: https://mcpjam-mintlify-docs-update-pr-4053-1786992934526.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # EvalTest > API reference for EvalTest The `EvalTest` class runs a single test scenario multiple times and provides statistical metrics like accuracy, precision, and recall. ## Import ```typescript theme={"theme":"css-variables"} import { EvalTest } from "@mcpjam/sdk"; ``` ## Constructor ```typescript theme={"theme":"css-variables"} new EvalTest(options: EvalTestConfig) ``` ### Parameters Configuration for the evaluation test. ### EvalTestConfig | Property | Type | Required | Description | | ------------------- | ------------------------ | -------- | ------------------------------------------------------------------------------ | | `id` | `string` | Yes | The case's identity. **Required from 5.0** — see below. | | `name` | `string` | Yes | Human-readable display name for the test | | `test` | `TestFunction` | Yes | The test function to run | | `expectedToolCalls` | `EvalExpectedToolCall[]` | No | Tool calls the iteration must make. **Enforced locally from 3.0** — see below. | | `matchOptions` | `EvalMatchOptions` | No | How `expectedToolCalls` is matched (order, arguments, extras) | | `predicates` | `Predicate[]` | No | Deterministic transcript checks that gate each iteration | **New in 5.0: `id` is required.** A case's identity used to be its `name`, so renaming a test forked its history — the run page started a fresh series and the old one was orphaned. `id` separates the two: history joins on `id`, and `name` is free to change. Mint one **once** and commit the literal beside the test: ```typescript theme={"theme":"css-variables"} import { mintCaseId } from "@mcpjam/sdk/contract"; console.log(mintCaseId()); // c_V1StGXR8Z5jdHi6Bmy — paste this in ``` Any URL-safe string of 1–128 characters (`A-Z a-z 0-9 _ -`) is accepted, so an id you already have — a hosted case id, a ticket number, a slug you maintain — works as-is; the `c_` prefix is only a grep convenience. Do **not** call `mintCaseId()` inline: an id regenerated on every run is not an identity. Constructing an `EvalTest` without an `id` throws, and the error carries a freshly minted id to paste. **Migrating from `externalCaseId`:** If your test already declares `externalCaseId`, reuse that value verbatim as `id` — do not mint a new one. The hosted history for that case is keyed on `externalCaseId`, so setting `id` to the same value keeps the history intact. Setting `id` to a different value is a hard error — at construction from 6.0, and at ingest. If your `externalCaseId` is not a valid id (it was never charset-bound, so it may contain spaces, slashes or punctuation), no `id` can agree with it. Rename the external id itself to a conforming value and declare that same value in both fields. ```typescript theme={"theme":"css-variables"} // Before new EvalTest({ externalCaseId: "case_123", name: "refunds a duplicate charge", ... }) // After — reuse the existing externalCaseId as id new EvalTest({ id: "case_123", externalCaseId: "case_123", name: "refunds a duplicate charge", ... }) ``` **New in 5.0: step objects reject unknown fields.** `prompt`, `toolCall`, `interact`, and `assert` steps — and their nested sub-objects such as `elementLocator` and widget assertions — are now **closed schemas**. A step carrying a field the contract does not declare fails to parse instead of silently dropping that field. Two things are deliberately left open: * `toolCallStep.arguments` — the tool's own argument object, whose keys come from the server's input schema. * A `Predicate` inside an `assert` step — predicates are a separate contract module with their own versioning. If you generate steps programmatically (for example, from an LLM or an importer), remove any fields that are not part of the step contract before passing them to the SDK. **Changed in 3.0.** `expectedToolCalls` used to be reporting metadata only: the local verdict came from your `test` function alone, while the dashboard recomputed the match — so the same run could show `accuracy() === 1` locally and fail in MCPJam. It is now enforced during the run, which means a test whose expectations were never actually checked can start failing on upgrade. If that happens, the expectation was wrong or over-specified. Fix it, relax it with `matchOptions` (e.g. `{ argumentMatching: "ignore" }`), or drop `expectedToolCalls` if it was only ever documentation. Tests that declare neither `expectedToolCalls` nor `predicates` behave exactly as before. #### matchOptions Layered suite → case, and validated when the test is constructed rather than mid-run. | Option | Values | Default | Meaning | | ------------------- | ---------------------------------------- | ----------- | -------------------------------------------- | | `toolCallOrder` | `"ignore"` \| `"strict"` \| `"superset"` | `"ignore"` | Whether expected calls must appear in order | | `argumentMatching` | `"partial"` \| `"exact"` \| `"ignore"` | `"partial"` | How strictly arguments must match | | `maxExtraToolCalls` | `number` \| `null` | `null` | Cap on unexpected calls (`null` = unlimited) | #### predicates Deterministic, state-based checks evaluated against the iteration transcript — same transcript, same verdict, which is what makes them usable as a CI gate. An iteration passes only if **every** predicate passes, independently of `failOnToolError`. Verdicts are reported under `metadata.predicates`, so the dashboard's check chips work for code-first runs too. See [eval reporting](/sdk/reference/eval-reporting#predicate-gate) for the full list of predicate types. ```typescript theme={"theme":"css-variables"} const test = new EvalTest({ id: "c_books_the_venue", name: "books the venue", expectedToolCalls: [{ toolName: "book_venue" }], matchOptions: { toolCallOrder: "strict" }, predicates: [ { type: "noToolErrors" }, { type: "responseContains", needle: "confirmed" }, ], test: async (executor) => { const r = await executor.run("Book the venue for Friday."); return r.hasToolCall("book_venue"); }, }); ``` The widget predicates (`widgetRendered`, `widgetRenderLatencyUnder`, `widgetNoConsoleErrors`) need render observations only a hosted run captures, so `EvalTest` rejects them at construction instead of failing every iteration. ### TestFunction Type ```typescript theme={"theme":"css-variables"} type TestFunction = (executor: HostExecutor) => boolean | Promise; ``` The test function receives a [`HostExecutor`](/sdk/reference/host-runner) (the interface implemented by both `HostRunner` and `HostRuntime`) and must return a `boolean`: * `true` = test passed * `false` = test failed Both `HostRunner`, `HostRuntime`, and mock executors implement the `HostExecutor` interface, so you can use any of them for testing. ### Example ```typescript theme={"theme":"css-variables"} const test = new EvalTest({ id: "c_addition_accuracy", name: "addition-accuracy", test: async (agent) => { const result = await agent.run("Add 2 and 3"); return result.hasToolCall("add"); }, }); ``` *** ## Methods ### run() Executes the test multiple times and returns detailed results. ```typescript theme={"theme":"css-variables"} run(executor: HostExecutor, options: EvalTestRunOptions): Promise ``` #### Parameters | Parameter | Type | Description | | ---------- | -------------------- | ---------------------------------------------------------------- | | `executor` | `HostExecutor` | The executor to test with (`HostRunner`, `HostRuntime`, or mock) | | `options` | `EvalTestRunOptions` | Run configuration | #### EvalTestRunOptions | Property | Type | Required | Default | Description | | ------------- | -------------------------------------------------------- | -------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `iterations` | `number` | Yes | - | Number of test runs | | `concurrency` | `number` | No | `5` | Parallel test runs | | `retries` | `number` | No | `0` | Retry failed tests | | `timeoutMs` | `number` | No | `30000` | Per-iteration wall-clock timeout in ms. The active prompt is aborted at this deadline, then given a 1 second grace period to settle so partial tool calls and trace history can still be captured. | | `onProgress` | `ProgressCallback` | No | - | Progress callback | | `onFailure` | `(report: string) => void` | No | - | Called with a failure report if any iterations fail | | `mcpjam` | [`MCPJamReportingConfig`](/sdk/reference/eval-reporting) | No | - | Auto-save results to MCPJam | Results are automatically saved to MCPJam after the run completes when an API key is available via `mcpjam.apiKey` or the `MCPJAM_API_KEY` environment variable. Set `mcpjam.enabled: false` to disable. #### ProgressCallback Type ```typescript theme={"theme":"css-variables"} type ProgressCallback = (completed: number, total: number) => void; ``` #### Example ```typescript theme={"theme":"css-variables"} await test.run(agent, { iterations: 30, concurrency: 5, retries: 2, timeoutMs: 30000, mcpjam: { suiteName: "Addition Eval", strict: false, }, onProgress: (done, total) => { console.log(`${done}/${total}`); }, onFailure: (report) => { console.error(report); }, }); ``` *** ### accuracy() Returns the success rate (0.0 - 1.0). ```typescript theme={"theme":"css-variables"} accuracy(): number ``` #### Returns `number` - Proportion of tests that passed. #### Example ```typescript theme={"theme":"css-variables"} console.log(`Accuracy: ${(test.accuracy() * 100).toFixed(1)}%`); // "Accuracy: 96.7%" ``` *** ### precision() Returns the precision metric, micro-averaged over the run's tool-call matches. ```typescript theme={"theme":"css-variables"} precision(): number ``` #### Returns `number` — True positives / (True positives + False positives). Counted per iteration from the expected/actual tool calls: a matched expectation is a true positive, an unexpected call is a false positive, a missing one is a false negative, and an argument mismatch counts as both. **Changed in 3.0.** `precision()`, `recall()` and `truePositiveRate()` all used to `return this.accuracy()` — three names for one number. They now compute real values, and **throw** when no test in the run declared `expectedToolCalls`, because there is nothing to compute them from. If you were reading `precision()` as a stand-in for accuracy, call [`accuracy()`](#accuracy) directly. *** ### recall() Returns the recall metric, micro-averaged over the run's tool-call matches. ```typescript theme={"theme":"css-variables"} recall(): number ``` #### Returns `number` — True positives / (True positives + False negatives). Throws when the run declared no `expectedToolCalls`. *** ### truePositiveRate() Returns the true positive rate (same as recall). ```typescript theme={"theme":"css-variables"} truePositiveRate(): number ``` *** ### unexpectedToolCallRate() The fraction of expectation-bearing iterations that made at least one tool call nobody asked for. ```typescript theme={"theme":"css-variables"} unexpectedToolCallRate(): number ``` #### Returns `number` — Iterations with an extra call / iterations that had expectations. `0` when the run declared no expectations. *** ### falsePositiveRate() **Deprecated in 3.0** — use [`unexpectedToolCallRate()`](#unexpectedtoolcallrate). The old implementation returned `failures / iterations`, which is the failure rate, not a false-positive rate. For runs with no `expectedToolCalls` it still returns that legacy value so existing dashboards do not change; for runs with expectations it now delegates to `unexpectedToolCallRate()`. ```typescript theme={"theme":"css-variables"} falsePositiveRate(): number ``` *** ### averageTokenUse() Returns the average tokens used per iteration. ```typescript theme={"theme":"css-variables"} averageTokenUse(): number ``` #### Returns `number` - Mean token count. #### Example ```typescript theme={"theme":"css-variables"} console.log(`Avg tokens: ${test.averageTokenUse()}`); ``` *** ### getResults() Returns the full run result from the last run. ```typescript theme={"theme":"css-variables"} getResults(): EvalRunResult | null ``` #### Returns `EvalRunResult | null` - The run result, or `null` if `run()` hasn't been called. #### EvalRunResult Type | Property | Type | Description | | ------------------ | ------------------- | ------------------------------------------ | | `iterations` | `number` | Total iterations run | | `successes` | `number` | Number that passed | | `failures` | `number` | Number that failed | | `results` | `boolean[]` | Pass/fail per iteration | | `iterationDetails` | `IterationResult[]` | Detailed per-iteration results | | `tokenUsage` | `object` | Aggregate and per-iteration token usage | | `latency` | `object` | Latency stats (e2e, llm, mcp) with p50/p95 | #### IterationResult Type | Property | Type | Description | | ------------ | ----------------------------- | ------------------------------------ | | `passed` | `boolean` | Whether this iteration passed | | `latencies` | `LatencyBreakdown[]` | Latency per prompt in this iteration | | `tokens` | `{ total, input, output }` | Token usage | | `error` | `string \| undefined` | Error message if failed | | `retryCount` | `number \| undefined` | Number of retries attempted | | `prompts` | `PromptResult[] \| undefined` | Prompt results from this iteration | *** ### getName() Returns the test's name. ```typescript theme={"theme":"css-variables"} getName(): string ``` *** ### getConfig() Returns the test's configuration. ```typescript theme={"theme":"css-variables"} getConfig(): EvalTestConfig ``` *** ### getAllIterations() Returns all iteration details from the last run. ```typescript theme={"theme":"css-variables"} getAllIterations(): IterationResult[] ``` *** ### getFailedIterations() Returns only the failed iterations from the last run. ```typescript theme={"theme":"css-variables"} getFailedIterations(): IterationResult[] ``` #### Example ```typescript theme={"theme":"css-variables"} const failures = test.getFailedIterations(); console.log(`${failures.length} failures`); for (const fail of failures) { console.log(` Error: ${fail.error}`); } ``` *** ### getSuccessfulIterations() Returns only the successful iterations from the last run. ```typescript theme={"theme":"css-variables"} getSuccessfulIterations(): IterationResult[] ``` *** ### getFailureReport() Returns a formatted failure report with traces from all failed iterations. Useful for debugging. ```typescript theme={"theme":"css-variables"} getFailureReport(): string ``` #### Example ```typescript theme={"theme":"css-variables"} await test.run(agent, { iterations: 30 }); if (test.accuracy() < 0.9) { console.error(test.getFailureReport()); } ``` *** ## Properties ### name The test's identifier (via `getName()`). ```typescript theme={"theme":"css-variables"} test.getName(); // "addition-accuracy" ``` *** ## Test Function Patterns ### Simple Tool Check ```typescript theme={"theme":"css-variables"} test: async (agent) => { const result = await agent.run("Add 5 and 3"); return result.hasToolCall("add"); }; ``` ### Argument Validation ```typescript theme={"theme":"css-variables"} test: async (agent) => { const result = await agent.run("Add 10 and 20"); const args = result.getToolArguments("add"); return args?.a === 10 && args?.b === 20; }; ``` ### Response Content ```typescript theme={"theme":"css-variables"} test: async (agent) => { const result = await agent.run("What is 5 + 5?"); return result.getText().includes("10"); }; ``` ### Multiple Conditions ```typescript theme={"theme":"css-variables"} test: async (agent) => { const result = await agent.run("Calculate 5 * 3"); return ( result.hasToolCall("multiply") && !result.hasError() && result.getText().length > 0 ); }; ``` ### Multi-Turn Conversation ```typescript theme={"theme":"css-variables"} test: async (agent) => { const r1 = await agent.run("Create a project"); const r2 = await agent.run("Add a task to it", { context: r1 }); return r1.hasToolCall("createProject") && r2.hasToolCall("createTask"); }; ``` ### With Validators ```typescript theme={"theme":"css-variables"} import { matchToolCallWithArgs } from "@mcpjam/sdk"; test: async (agent) => { const result = await agent.run("Add 2 and 3"); return matchToolCallWithArgs("add", { a: 2, b: 3 }, result.getToolCalls()); }; ``` *** ## Complete Example ```typescript theme={"theme":"css-variables"} import { MCPClientManager, HostRunner, EvalTest } from "@mcpjam/sdk"; async function main() { const manager = new MCPClientManager({ everything: { command: "npx", args: ["-y", "@modelcontextprotocol/server-everything"], }, }); await manager.connectToServer("everything"); const agent = new HostRunner({ tools: await manager.getTools(), model: "anthropic/claude-sonnet-5", apiKey: process.env.ANTHROPIC_API_KEY, temperature: 0.1, }); const test = new EvalTest({ id: "c_addition", name: "addition", test: async (agent) => { const r = await agent.run("Add 2 and 3"); return r.hasToolCall("add"); }, }); console.log("Running evaluation..."); const result = await test.run(agent, { iterations: 30, concurrency: 5, mcpjam: { suiteName: "Addition Eval" }, onProgress: (done, total) => { process.stdout.write(`\r${done}/${total}`); }, onFailure: (report) => { console.error(report); }, }); console.log("\n\nResults:"); console.log(` Accuracy: ${(test.accuracy() * 100).toFixed(1)}%`); console.log(` Precision: ${(test.precision() * 100).toFixed(1)}%`); console.log(` Recall: ${(test.recall() * 100).toFixed(1)}%`); console.log(` Avg tokens: ${test.averageTokenUse()}`); console.log(` Iterations: ${result.iterations}`); console.log(` Successes: ${result.successes}`); await manager.disconnectServer("everything"); } ``` *** ## Related * [Running Evals](/sdk/concepts/running-evals) - Conceptual guide * [EvalSuite Reference](/sdk/reference/eval-suite) - Group multiple tests * [Saving Eval Results](/sdk/reference/eval-reporting) - Save results to MCPJam * [Validators Reference](/sdk/reference/validators) - Assertion functions