Interface: TestContext

The t context passed to an eval's test function.

Properties

expected

readonly expected: Record<string, unknown> | undefined;

The current dataset row's expectations (ground truth / guidelines), or undefined when the row has none or the eval isn't dataset-driven.

input

readonly input: Record<string, unknown>;

The current dataset row's inputs when the eval is dataset-driven (see EvalDefinition.dataset); {} for a plain single-run eval.

judge

judge: {
  closedQA: Promise<AssertionHandle>;
  custom: Promise<AssertionHandle>;
  factuality: Promise<AssertionHandle>;
};

LLM-as-judge scoring of the last reply (via autoevals → a Databricks judge model). Each returns a scored assertion that gates by default (a miss fails the eval); chain .atLeast(n) to change the pass threshold or .soft() to demote to a tracked-only metric. Requires the judge to be configured (--judge-model).

closedQA()

closedQA(criteria: string): Promise<AssertionHandle>;

Score whether the reply answers the question, per optional criteria.

Parameters

Returns

Promise<AssertionHandle>

custom()

custom(spec: CustomJudgeSpec): Promise<AssertionHandle>;

A custom prompt-template judge (the TS analog of MLflow's @scorer).

Parameters

Returns

Promise<AssertionHandle>

factuality()

factuality(expected: string): Promise<AssertionHandle>;

Score factuality of the reply against an expected reference.

Parameters

Returns

Promise<AssertionHandle>

reply

readonly reply: string;

The last assistant reply.

sessionId

readonly sessionId: string | undefined;

The current session/thread id, if any.

toolCalls

readonly toolCalls: string[];

Tools called during the last turn.

Methods

calledTool()

calledTool(name: string): AssertionHandle;

Assert a tool was called during the run (gate by default).

Parameters

Returns

AssertionHandle

calledToolWith()

calledToolWith(name: string, expected: Record<string, unknown>): AssertionHandle;

Assert a tool was called with arguments that deep-contain expected: every key in expected must equal the actual argument (recursively for nested objects; arrays match element-for-element), so extra arguments are ignored. Gate by default.

Parameters

Returns

AssertionHandle

check()

check(value: string, matcher: Matcher): AssertionHandle;

Assert a value against a matcher, e.g. t.check(t.reply, includes("Sunny")).

Parameters

Returns

AssertionHandle

reset()

reset(): void;

Start a fresh conversation: the next send opens a new thread with no history. Use to run several independent one-shot checks in one test. Consecutive sends (without a reset) stay in one multi-turn conversation.

Returns

void

send()

send(message: string): Promise<void>;

Send a user message to the agent and capture its response.

Parameters

Returns

Promise<void>

skip()

skip(reason?: string): never;

Skip this eval with an optional reason.

Parameters

Returns

never

succeeded()

succeeded(): AssertionHandle;

Assert the last turn completed successfully (gate by default).

Returns

AssertionHandle