Package com.yahoo.schema.expressiontransforms

Class TokenTransformer

java.lang.Object
com.yahoo.searchlib.rankingexpression.transform.ExpressionTransformer<RankProfileTransformContext>
com.yahoo.schema.expressiontransforms.TokenTransformer
public class TokenTransformer extends com.yahoo.searchlib.rankingexpression.transform.ExpressionTransformer<RankProfileTransformContext>
Convenience feature transforms for inputs to Transformer type models. Replaces features of the form tokenInputIds tokenTypeIds tokenAttentionMask to tensor generation expressions that generate the required input. In general, these models expect input of the form: CLS + arg1 + SEP + arg2 + SEP + 0's
Author:
lesters

  • Constructor Summary

    Constructors
    Constructor
    Description
    TokenTransformer()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static com.yahoo.tensor.TensorType
    createTensorType(String featureName, com.yahoo.searchlib.rankingexpression.rule.ExpressionNode argument)
     
    com.yahoo.searchlib.rankingexpression.rule.ExpressionNode
    transform(com.yahoo.searchlib.rankingexpression.rule.ExpressionNode node, RankProfileTransformContext context)
     

    Methods inherited from class com.yahoo.searchlib.rankingexpression.transform.ExpressionTransformer

    transform, transformChildren

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • TokenTransformer

      public TokenTransformer()

  • Method Details

    • transform

      public com.yahoo.searchlib.rankingexpression.rule.ExpressionNode transform(com.yahoo.searchlib.rankingexpression.rule.ExpressionNode node, RankProfileTransformContext context)
      Specified by:
      transform in class com.yahoo.searchlib.rankingexpression.transform.ExpressionTransformer<RankProfileTransformContext>

    • createTensorType

      public static com.yahoo.tensor.TensorType createTensorType(String featureName, com.yahoo.searchlib.rankingexpression.rule.ExpressionNode argument)