Data

The data module loads datasets for evaluations. It supports inline arrays, Hugging Face Hub datasets, and URLs, and can return either plain arrays or Arrow tables.

Key types & signatures

type DatasetRef = unknown[] | { url: string } | { hf: string; split?: string; revision?: string };

function loadDataset(ref: DatasetRef, options?: DatasetLoadOptions): Promise<unknown[]>;

function loadDatasetFromHub(
  repoId: string,
  options?: DatasetLoadOptions & { split?: string; revision?: string }
): Promise<unknown[]>;

type ArrowTable = import('apache-arrow').Table;

Load a dataset from the Hub

import { loadDatasetFromHub } from '@dreadnode/agents';

const rows = await loadDatasetFromHub('dreadnode/evals-support', {
  split: 'train',
});

console.log(rows.length);

Load an inline dataset and keep Arrow format

import { loadDatasetAsArrow, type ArrowTable } from '@dreadnode/agents';

const table: ArrowTable = await loadDatasetAsArrow([
  { input: 'What is Dreadnode?', expected: 'An AI agent platform.' },
  { input: 'What is a generator?', expected: 'A model wrapper.' },
]);

console.log(table.numRows);

Use loadDataset in evaluations

import { loadDataset } from '@dreadnode/agents';

const samples = await loadDataset({
  hf: 'dreadnode/evals-support',
  split: 'test',
});

for (const sample of samples) {
  console.log(sample);
}