Package dev.braintrust.eval

Interface Dataset<INPUT,OUTPUT>

All Known Implementing Classes:

DatasetBrainstoreImpl

public interface Dataset<INPUT,OUTPUT>

Datasets define the cases for evals. This interface provides a means of iterating through all cases of a particular dataset.

The most common implementations are in-memory datasets, and datasets fetched from the Braintrust API.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static interface	Dataset.Cursor<CASE>

Method Summary

Modifier and Type	Method	Description
static <INPUT, OUTPUT> Dataset<INPUT,OUTPUT>	fetchFromBraintrust(BraintrustApiClient apiClient, String projectName, String datasetName, String datasetVersion)	Deprecated.
static <INPUT, OUTPUT> Dataset<INPUT,OUTPUT>	fetchFromBraintrust(BraintrustOpenApiClient apiClient, String projectName, String datasetName, String datasetVersion)	Deprecated. the INPUT and OUTPUT type parameters are not applied at runtime: case values are returned as raw JSON-decoded objects (e.g.
static <INPUT, OUTPUT> Dataset<INPUT,OUTPUT>	fetchFromBraintrust(BraintrustOpenApiClient apiClient, String projectName, String datasetName, String datasetVersion, Class<INPUT> inputClass, Class<OUTPUT> outputClass)	Fetch a dataset from Braintrust, deserializing each case's input and expected values into the given types using the SDK's shared Jackson mapper (see BraintrustJsonMapper).
static <INPUT, OUTPUT> Dataset<INPUT,OUTPUT>	fetchFromBraintrust(BraintrustOpenApiClient apiClient, String projectName, String datasetName, String datasetVersion, Function<Object,INPUT> inputConverter, Function<Object,OUTPUT> outputConverter)	Fetch a dataset from Braintrust, converting each case's input and expected values with the supplied converter functions.
default void	forEach(Consumer<DatasetCase<INPUT,OUTPUT>> consumer)	Convenience method to safely iterate all items in a dataset.
String	id()
static <INPUT, OUTPUT> Dataset<INPUT,OUTPUT>	of(DatasetCase<INPUT,OUTPUT>... cases)	Create an in-memory Dataset containing the provided cases.
Dataset.Cursor<DatasetCase<INPUT,OUTPUT>>	openCursor()
Optional<String>	version()	Dataset version.

Method Details

openCursor

Dataset.Cursor<DatasetCase<INPUT,OUTPUT>> openCursor()

id

String id()

version

Optional<String> version()

Dataset version. Empty means the dataset will fetch latest upon every cursor open

forEach

default void forEach(Consumer<DatasetCase<INPUT,OUTPUT>> consumer)

Convenience method to safely iterate all items in a dataset.

of

@SafeVarargs
static <INPUT,
OUTPUT> Dataset<INPUT,OUTPUT> of(DatasetCase<INPUT,OUTPUT>... cases)

Create an in-memory Dataset containing the provided cases.

fetchFromBraintrust

@Deprecated
static <INPUT,
OUTPUT> Dataset<INPUT,OUTPUT> fetchFromBraintrust(BraintrustApiClient apiClient,
 String projectName,
 String datasetName,
 @Nullable
 String datasetVersion)

Deprecated.

fetchFromBraintrust

@Deprecated
static <INPUT,
OUTPUT> Dataset<INPUT,OUTPUT> fetchFromBraintrust(BraintrustOpenApiClient apiClient,
 String projectName,
 String datasetName,
 @Nullable
 String datasetVersion)

Deprecated.

the INPUT and OUTPUT type parameters are not applied at runtime: case values are returned as raw JSON-decoded objects (e.g. LinkedHashMap). Use fetchFromBraintrust(BraintrustOpenApiClient, String, String, String, Class, Class) instead.

Fetch a dataset from Braintrust.

fetchFromBraintrust

static <INPUT,
OUTPUT> Dataset<INPUT,OUTPUT> fetchFromBraintrust(BraintrustOpenApiClient apiClient,
 String projectName,
 String datasetName,
 @Nullable
 String datasetVersion,
 Class<INPUT> inputClass,
 Class<OUTPUT> outputClass)

Fetch a dataset from Braintrust, deserializing each case's input and expected values into the given types using the SDK's shared Jackson mapper (see BraintrustJsonMapper).

Parameters:

apiClient - the Braintrust API client

projectName - the project containing the dataset

datasetName - the name of the dataset within the project

datasetVersion - the dataset version to pin, or null to fetch the latest version upon every cursor open

inputClass - the type to deserialize each case's input into

outputClass - the type to deserialize each case's expected into

fetchFromBraintrust

static <INPUT,
OUTPUT> Dataset<INPUT,OUTPUT> fetchFromBraintrust(BraintrustOpenApiClient apiClient,
 String projectName,
 String datasetName,
 @Nullable
 String datasetVersion,
 Function<Object,INPUT> inputConverter,
 Function<Object,OUTPUT> outputConverter)

Fetch a dataset from Braintrust, converting each case's input and expected values with the supplied converter functions. Converters receive the raw JSON-decoded value (typically a Map<String, Object>, List, String, Number, Boolean, or null) and must tolerate null.

Parameters:

apiClient - the Braintrust API client

projectName - the project containing the dataset

datasetName - the name of the dataset within the project

datasetVersion - the dataset version to pin, or null to fetch the latest version upon every cursor open

inputConverter - converts each case's raw input value

outputConverter - converts each case's raw expected value