Package dev.braintrust.eval

Interface Scorer<INPUT,OUTPUT>

Type Parameters:
INPUT - type of the input data
OUTPUT - type of the output data
All Known Subinterfaces:
TracedScorer<INPUT,OUTPUT>
All Known Implementing Classes:
ScorerBrainstoreImpl
public interface Scorer<INPUT,OUTPUT>
A scorer evaluates the result of a task against a dataset case, producing a score between 0 (inclusive) and 1 (inclusive).

  • Method Details

    • getName

      String getName()

    • score

      List<Score> score(TaskResult<INPUT,OUTPUT> taskResult)
      Scores the result of a successful task execution.
      Parameters:
      taskResult - the task output and originating dataset case
      Returns:
      one or more scores, each with a value between 0 and 1 inclusive

      If this method thows, the error will be recorded on the span and scoreForScorerException(java.lang.Exception, dev.braintrust.eval.TaskResult<INPUT, OUTPUT>) will be called as a fallback

    • scoreForTaskException

      default List<Score> scoreForTaskException(Exception taskException, DatasetCase<INPUT,OUTPUT> datasetCase)
      Provides fallback scores when the task function threw an exception. Called instead of score(dev.braintrust.eval.TaskResult<INPUT, OUTPUT>) for each scorer.
      Parameters:
      taskException - the exception thrown by the task
      datasetCase - the dataset case that was being evaluated
      Returns:
      fallback scores, or an empty list to skip scoring for this case

    • scoreForScorerException

      default List<Score> scoreForScorerException(Exception scorerException, TaskResult<INPUT,OUTPUT> taskResult)
      Provides fallback scores when this scorer's score(dev.braintrust.eval.TaskResult<INPUT, OUTPUT>) method threw an exception.
      Parameters:
      scorerException - the exception thrown by score(dev.braintrust.eval.TaskResult<INPUT, OUTPUT>)
      taskResult - the task result that was being scored
      Returns:
      fallback scores, or an empty list to skip scoring for this case

    • of

      static <INPUT, OUTPUT> Scorer<INPUT,OUTPUT> of(String scorerName, Function<TaskResult<INPUT,OUTPUT>,Double> scorerFn)

    • of

      static <INPUT, OUTPUT> Scorer<INPUT,OUTPUT> of(String scorerName, BiFunction<OUTPUT,OUTPUT,Double> scorerFn)

    • fetchFromBraintrust

      @Deprecated static <INPUT, OUTPUT> Scorer<INPUT,OUTPUT> fetchFromBraintrust(BraintrustApiClient apiClient, String projectName, String scorerSlug, @Nullable String version)
      Deprecated.

    • fetchFromBraintrust

      static <INPUT, OUTPUT> Scorer<INPUT,OUTPUT> fetchFromBraintrust(BraintrustOpenApiClient apiClient, String projectName, String scorerSlug, @Nullable String version)
      Fetch a scorer from Braintrust by project name and slug.
      Parameters:
      apiClient - the API client to use
      projectName - the name of the project containing the scorer
      scorerSlug - the unique slug identifier for the scorer
      version - optional version of the scorer to fetch
      Returns:
      a Scorer that invokes the remote function
      Throws:
      RuntimeException - if the scorer is not found

    • fetchFromBraintrust

      static <INPUT, OUTPUT> Scorer<INPUT,OUTPUT> fetchFromBraintrust(BraintrustOpenApiClient apiClient, String projectName, String scorerSlug, @Nullable String version, Function<INPUT,Object> inputConverter, Function<OUTPUT,Object> outputConverter)
      Fetch a scorer from Braintrust by project name and slug, with custom converters that transform the input, output, and expected scorer argument values into a JSON-serializable form.

      By default (fetchFromBraintrust(BraintrustOpenApiClient, String, String, String)), argument values are serialized with the SDK's shared mapper (BraintrustJsonMapper). Use this variant to control serialization, e.g. with a custom ObjectMapper: 

      
 Scorer<MyInput, MyOutput> scorer = Scorer.fetchFromBraintrust(
         client, project, slug, null, myMapper::valueToTree, myMapper::valueToTree);

      Case metadata and eval parameters are always serialized with the OpenAPI client's mapper.

      Parameters:
      apiClient - the API client to use
      projectName - the name of the project containing the scorer
      scorerSlug - the unique slug identifier for the scorer
      version - optional version of the scorer to fetch
      inputConverter - converts each case's input value into a JSON-serializable form (e.g. a Jackson JsonNode, Map, or scalar); never invoked with null
      outputConverter - converts the task output and case expected values into a JSON-serializable form; never invoked with null
      Returns:
      a Scorer that invokes the remote function
      Throws:
      RuntimeException - if the scorer is not found