Package dev.braintrust.devserver

Class RemoteEval<INPUT,OUTPUT>

java.lang.Object
dev.braintrust.devserver.RemoteEval<INPUT,OUTPUT>
Type Parameters:
INPUT - The type of input data for the evaluation
OUTPUT - The type of output produced by the task
public class RemoteEval<INPUT,OUTPUT> extends Object
Represents a remote evaluator that can be exposed via the dev server.

  • Method Details

    • builder

      public static <INPUT, OUTPUT> RemoteEval.Builder<INPUT,OUTPUT> builder(Class<INPUT> inputClass, Class<OUTPUT> outputClass)
      Create a builder for an eval whose case input and expected values are deserialized into the given types using the SDK's shared Jackson mapper (see BraintrustJsonMapper). 
      
 var eval = RemoteEval.builder(MyInput.class, MyOutput.class)
         .name("my-eval")
         .taskFunction(input -> ...)
         .build();
      Parameters:
      inputClass - the type to deserialize each case's input into
      outputClass - the type to deserialize each case's expected into

    • builder

      public static <INPUT, OUTPUT> RemoteEval.Builder<INPUT,OUTPUT> builder(Function<Object,INPUT> inputConverter, Function<Object,OUTPUT> outputConverter)
      Create a builder for an eval whose case input and expected values are converted with the supplied functions. Converters receive the raw JSON-decoded value (typically a Map<String, Object>, List, String, Number, Boolean, or null) and must tolerate null. Most callers should prefer builder(Class, Class); use this variant for generic types or custom deserialization.
      Parameters:
      inputConverter - converts each case's raw input value
      outputConverter - converts each case's raw expected value

    • builder

      @Deprecated public static <INPUT, OUTPUT> RemoteEval.Builder<INPUT,OUTPUT> builder()
      Deprecated.
      the INPUT and OUTPUT type parameters are not applied at runtime, and the task/scorers will receive raw JSON objects, throwing ClassCastException unless those types are Object or Map. Use builder(Class, Class) or builder(Function, Function) instead.
      Create a builder with no input/expected deserialization: case values are passed through as raw JSON-decoded objects (e.g. LinkedHashMap).

    • getName

      @Nonnull public String getName()
      The name of this evaluator (used as identifier)

    • getTask

      @Nonnull public Task<INPUT,OUTPUT> getTask()
      The task function that performs the evaluation

      The task function must be thread safe.

    • getScorers

      @Nonnull public List<Scorer<INPUT,OUTPUT>> getScorers()
      List of scorers for this evaluator

      The score function must be thread safe.

    • getParameters

      @Nonnull public List<ParameterDef<?>> getParameters()
      Optional parameter definitions that can be configured from the UI

    • getInputConverter

      @Nonnull public Function<Object,INPUT> getInputConverter()
      Converts raw JSON-decoded input values from eval requests into INPUT. Defaults to an unchecked cast of the raw value.

    • getOutputConverter

      @Nonnull public Function<Object,OUTPUT> getOutputConverter()
      Converts raw JSON-decoded expected values from eval requests into OUTPUT. Defaults to an unchecked cast of the raw value.