Package dev.braintrust.eval

Interface Task<INPUT,OUTPUT>

Type Parameters:

INPUT - type of the input data

OUTPUT - type of the output data

public interface Task<INPUT,OUTPUT>

A task function that runs against a single dataset case and produces an output.

If the task throws an exception, the error is recorded on the span and each scorer's Scorer.scoreForTaskException(java.lang.Exception, dev.braintrust.eval.DatasetCase<INPUT, OUTPUT>) method is invoked instead of Scorer.score(dev.braintrust.eval.TaskResult<INPUT, OUTPUT>).

Tasks may optionally accept Parameters by overriding the two-arg apply(DatasetCase, Parameters) method. Tasks that don't need parameters can override the single-arg apply(DatasetCase) method instead — the two-arg version delegates to it by default.

Method Summary

Modifier and Type	Method	Description
default TaskResult<INPUT,OUTPUT>	apply(DatasetCase<INPUT,OUTPUT> datasetCase)	Executes this task against a single dataset case.
TaskResult<INPUT,OUTPUT>	apply(DatasetCase<INPUT,OUTPUT> datasetCase, Parameters parameters)	Executes this task against a single dataset case, with access to merged eval parameters.

Method Details

apply

TaskResult<INPUT,OUTPUT> apply(DatasetCase<INPUT,OUTPUT> datasetCase,
 Parameters parameters)
                        throws Exception

Executes this task against a single dataset case, with access to merged eval parameters.

Override this method if your task needs to read parameter values (e.g., model name, temperature). The default implementation delegates to apply(DatasetCase), ignoring parameters.

Parameters:

datasetCase - the dataset case to evaluate

parameters - the merged parameter values for this eval run

Returns:

the task result containing the output and the originating dataset case

Throws:

Exception - if the task fails, the error will be recorded on the span and scoring will fall back to Scorer.scoreForTaskException(java.lang.Exception, dev.braintrust.eval.DatasetCase<INPUT, OUTPUT>)

apply

default TaskResult<INPUT,OUTPUT> apply(DatasetCase<INPUT,OUTPUT> datasetCase)
                                throws Exception

Executes this task against a single dataset case.

Override this method for tasks that do not need parameter values. If you need parameters, override apply(DatasetCase, Parameters) instead.

Parameters:

datasetCase - the dataset case to evaluate

Returns:

the task result containing the output and the originating dataset case

Throws:

Exception - if the task fails, the error will be recorded on the span and scoring will fall back to Scorer.scoreForTaskException(java.lang.Exception, dev.braintrust.eval.DatasetCase<INPUT, OUTPUT>)