Interface: RunEvalsOptions

Properties

baseUrl

baseUrl: string;

Base URL of the running app to drive, e.g. http://localhost:3000.

concurrency?

optional concurrency: number;

Max evals to drive concurrently. Each eval opens one stream to the app as the same user, so keep this at or below the app's maxConcurrentStreamsPerUser (default 5) or the surplus streams hit the 429 guard. Defaults to 4; clamped to [1, total].

filter?

optional filter: string;

Substring filter on <agent>/<id> (or an exact agent id).

headers?

optional headers: Record<string, string>;

Extra request headers for the driver (e.g. auth for a deployed app).

judge?

optional judge: {
  host: string;
  model: string;
  token: string;
};

When set, enable t.judge.* LLM-as-judge scoring via autoevals against a Databricks serving endpoint (model).

host

host: string;

model

model: string;

token

token: string;

mlflow?

optional mlflow: {
  experimentId: string;
  host: string;
  sqlWarehouseId?: string;
  token: string;
};

When set, create a native MLflow "Evaluation run": each eval's trace is linked to the run, pass/fail is written as feedback, and aggregate metrics are logged. Requires Databricks creds + the target experiment.

experimentId

experimentId: string;

host

host: string;

sqlWarehouseId?

optional sqlWarehouseId: string;

SQL warehouse id for writing assessments to UC-backed (V4) traces.

token

token: string;

now?

optional now: number;

Wall-clock timestamp (ms) for run create/finish — pass Date.now().

onEvent()?

optional onEvent: (event: EvalProgress) => void;

Progress callback, invoked as evals are discovered, started, and finished.

Parameters

Returns

void

retries?

optional retries: number;

Re-run an eval up to this many extra times when it fails on infrastructure — a thrown error/timeout (result.error) or a transport/agent turn failure (result.infraFailure). Assertion failures are never retried. Defaults to 0.

rootDir?

optional rootDir: string;

Project root containing server/agents/. Defaults to process.cwd().

strict?

optional strict: boolean;

Soft assertion failures also fail the eval.

tags?

optional tags: string[];

Only run evals whose tags intersect this list. Empty/undefined runs all. Tags live on the eval def, so filtering happens after each file is loaded.

timeoutMs?

optional timeoutMs: number;

Default per-eval timeout (ms): runEval races the whole test against it and it also caps each driver turn. A per-eval def.timeoutMs overrides it, and it wins over an agent's evals.config.ts timeoutMs. Unbounded when unset.

warehouseId?

optional warehouseId: string;

SQL warehouse id used to read managed evaluation datasets.

workspaceClient?

optional workspaceClient: WorkspaceClient;

Workspace client used to read managed evaluation datasets (for evals that declare dataset). Required alongside warehouseId for those evals.