Scorers

Scorers turn outputs into metrics. Use built-in scorers whenever possible before writing custom logic so you stay consistent across evaluations and hooks.

Key types & signatures

type ScorerFn<T = unknown> = (
  obj: T
) => number | boolean | Metric | Promise<number | boolean | Metric>;

class Scorer<T = unknown> {
  constructor(options: { name: string; fn: ScorerFn<T>; autoIncrementStep?: boolean });
  score(obj: T): Promise<Metric>;
}

function scorer<T = unknown>(
  name: string,
  fn: ScorerFn<T>,
  options?: { autoIncrementStep?: boolean }
): Scorer<T>;

function condition<T = unknown>(
  fn: (obj: T) => boolean | Promise<boolean>,
  options?: { catchErrors?: boolean; defaultValue?: boolean }
): Condition<T>;

Built-in scorers

Prefer these before writing custom logic:

llmJudge — LLM-based scoring for subjective criteria
similarity — Compare two strings by semantic or lexical similarity
contains — Check for required substrings or tokens
condition — Convert a predicate into a scoring condition

Use built-in scorers

import { llmJudge, similarity, contains } from '@dreadnode/agents';
import { createGenerator } from '@dreadnode/agents';
import { anthropic } from '@ai-sdk/anthropic';

const generator = createGenerator(anthropic('claude-sonnet-4-20250514'));

const judge = llmJudge({
  generator,
  criteria: 'Evaluate clarity and factual accuracy.',
});

const similarityScore = similarity({ reference: 'Dreadnode is an AI agent platform.' });
const mustMention = contains({ pattern: 'agent' });

const output = 'Dreadnode is a platform for building AI agents.';
const judgeMetric = await judge.score(output);
const simMetric = await similarityScore.score(output);
const containsMetric = await mustMention.score(output);

console.log(judgeMetric.value, simMetric.value, containsMetric.value);

Build a custom scorer

import { Scorer } from '@dreadnode/agents';

const lengthScorer = new Scorer({
  name: 'length_bonus',
  fn: (text: string) => (text.length > 120 ? 1 : 0),
});

const metric = await lengthScorer.score('Short response.');
console.log(metric.value);