Class

com.github.jelmerk.spark.knn.bruteforce

BruteForceSimilarity

Related Doc: package bruteforce

Permalink

class BruteForceSimilarity extends KnnAlgorithm[BruteForceSimilarityModel]

Nearest neighbor search using a brute force approach. This will be very slow. It is in most cases not recommended for production use. But can be used to determine the accuracy of an approximative index.

Linear Supertypes
KnnAlgorithm[BruteForceSimilarityModel], KnnAlgorithmParams, KnnModelParams, HasPredictionCol, HasFeaturesCol, Estimator[BruteForceSimilarityModel], PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. BruteForceSimilarity
  2. KnnAlgorithm
  3. KnnAlgorithmParams
  4. KnnModelParams
  5. HasPredictionCol
  6. HasFeaturesCol
  7. Estimator
  8. PipelineStage
  9. Logging
  10. Params
  11. Serializable
  12. Serializable
  13. Identifiable
  14. AnyRef
  15. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new BruteForceSimilarity()

    Permalink
  2. new BruteForceSimilarity(uid: String)

    Permalink

    uid

    identifier

Type Members

  1. type TIndex[TId, TVector, TItem <: Item[TId, TVector], TDistance] = BruteForceIndex[TId, TVector, TItem, TDistance]

    Permalink

    Type of index.

    Type of index.

    Attributes
    protected
    Definition Classes
    BruteForceSimilarity → KnnAlgorithm

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  4. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  6. final def clear(param: Param[_]): BruteForceSimilarity.this.type

    Permalink
    Definition Classes
    Params
  7. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  8. def copy(extra: ParamMap): Estimator[BruteForceSimilarityModel]

    Permalink
    Definition Classes
    KnnAlgorithm → Estimator → PipelineStage → Params
  9. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  10. def createIndex[TId, TVector, TItem <: Item[TId, TVector] with Product, TDistance](dimensions: Int, maxItemCount: Int, distanceFunction: DistanceFunction[TVector, TDistance])(implicit distanceOrdering: Ordering[TDistance], idSerializer: ObjectSerializer[TId], itemSerializer: ObjectSerializer[TItem]): BruteForceIndex[TId, TVector, TItem, TDistance]

    Permalink

    Create the index used to do the nearest neighbor search.

    Create the index used to do the nearest neighbor search.

    TId

    type of the index item identifier

    TVector

    type of the index item vector

    TItem

    type of the index item

    TDistance

    type of distance between items

    dimensions

    dimensionality of the items stored in the index

    maxItemCount

    maximum number of items the index can hold

    distanceFunction

    the distance function

    distanceOrdering

    the distance ordering

    idSerializer

    invoked for serializing ids when saving the index

    itemSerializer

    invoked for serializing items when saving items

    returns

    create an index

    Attributes
    protected
    Definition Classes
    BruteForceSimilarity → KnnAlgorithm
  11. def createModel[TId, TVector, TItem <: Item[TId, TVector] with Product, TDistance](uid: String, outputDir: String, numPartitions: Int)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[TId], arg1: scala.reflect.api.JavaUniverse.TypeTag[TVector], arg2: scala.reflect.api.JavaUniverse.TypeTag[TItem], arg3: scala.reflect.api.JavaUniverse.TypeTag[TDistance], evId: ClassTag[TId], evVector: ClassTag[TVector], distanceNumeric: Numeric[TDistance]): BruteForceSimilarityModel

    Permalink

    Creates the model to be returned from fitting the data.

    Creates the model to be returned from fitting the data.

    TId

    type of the index item identifier

    TVector

    type of the index item vector

    TItem

    type of the index item

    TDistance

    type of distance between items

    uid

    identifier

    outputDir

    directory containing the persisted indices

    numPartitions

    number of index partitions

    returns

    model

    Attributes
    protected
    Definition Classes
    BruteForceSimilarity → KnnAlgorithm
  12. final def defaultCopy[T <: Params](extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  13. final val distanceFunction: Param[String]

    Permalink

    Param for the distance function to use.

    Param for the distance function to use. One of "bray-curtis", "canberra", "cosine", "correlation", "euclidean", "inner-product", "manhattan" or the fully qualified classname of a distance function Default: "cosine"

    Definition Classes
    KnnAlgorithmParams
  14. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  15. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  16. final val excludeSelf: BooleanParam

    Permalink

    Param that indicates whether to not return the a candidate when it's identifier equals the query identifier Default: false

    Param that indicates whether to not return the a candidate when it's identifier equals the query identifier Default: false

    Definition Classes
    KnnModelParams
  17. def explainParam(param: Param[_]): String

    Permalink
    Definition Classes
    Params
  18. def explainParams(): String

    Permalink
    Definition Classes
    Params
  19. final def extractParamMap(): ParamMap

    Permalink
    Definition Classes
    Params
  20. final def extractParamMap(extra: ParamMap): ParamMap

    Permalink
    Definition Classes
    Params
  21. final val featuresCol: Param[String]

    Permalink
    Definition Classes
    HasFeaturesCol
  22. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  23. def fit(dataset: Dataset[_]): BruteForceSimilarityModel

    Permalink
    Definition Classes
    KnnAlgorithm → Estimator
  24. def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): Seq[BruteForceSimilarityModel]

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  25. def fit(dataset: Dataset[_], paramMap: ParamMap): BruteForceSimilarityModel

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  26. def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): BruteForceSimilarityModel

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" ) @varargs()
  27. final def get[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  28. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  29. final def getDefault[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  30. final def getDistanceFunction: String

    Permalink

    Definition Classes
    KnnAlgorithmParams
  31. final def getExcludeSelf: Boolean

    Permalink

    Definition Classes
    KnnModelParams
  32. final def getFeaturesCol: String

    Permalink
    Definition Classes
    HasFeaturesCol
  33. final def getIdentifierCol: String

    Permalink

    Definition Classes
    KnnAlgorithmParams
  34. final def getK: Int

    Permalink

    Definition Classes
    KnnModelParams
  35. final def getNumPartitions: Int

    Permalink

    Definition Classes
    KnnAlgorithmParams
  36. final def getNumReplicas: Int

    Permalink

    Definition Classes
    KnnModelParams
  37. final def getOrDefault[T](param: Param[T]): T

    Permalink
    Definition Classes
    Params
  38. final def getOutputFormat: String

    Permalink

    Definition Classes
    KnnModelParams
  39. final def getParallelism: Int

    Permalink

    Definition Classes
    KnnModelParams
  40. def getParam(paramName: String): Param[Any]

    Permalink
    Definition Classes
    Params
  41. final def getPartitionCol: String

    Permalink

    Definition Classes
    KnnAlgorithmParams
  42. final def getPredictionCol: String

    Permalink
    Definition Classes
    HasPredictionCol
  43. final def getQueryIdentifierCol: String

    Permalink

    Definition Classes
    KnnModelParams
  44. final def getQueryPartitionsCol: String

    Permalink

    Definition Classes
    KnnModelParams
  45. final def getSimilarityThreshold: Double

    Permalink

    Definition Classes
    KnnModelParams
  46. final def hasDefault[T](param: Param[T]): Boolean

    Permalink
    Definition Classes
    Params
  47. def hasParam(paramName: String): Boolean

    Permalink
    Definition Classes
    Params
  48. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  49. final val identifierCol: Param[String]

    Permalink

    Param for the column name for the row identifier.

    Param for the column name for the row identifier. Default: "id"

    Definition Classes
    KnnAlgorithmParams
  50. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  51. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  52. final def isDefined(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  53. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  54. final def isSet(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  55. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  56. final val k: IntParam

    Permalink

    Param for number of neighbors to find (> 0).

    Param for number of neighbors to find (> 0). Default: 5

    Definition Classes
    KnnModelParams
  57. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  58. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  59. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  60. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  61. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  62. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  63. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  64. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  65. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  66. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  67. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  68. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  69. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  70. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  71. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  72. final val numPartitions: IntParam

    Permalink

    Number of partitions (default: 1)

    Number of partitions (default: 1)

    Definition Classes
    KnnAlgorithmParams
  73. final val numReplicas: IntParam

    Permalink

    Param that specifies the number of index replicas to create when querying the index.

    Param that specifies the number of index replicas to create when querying the index. More replicas means you can execute more queries in parallel at the expense of increased resource usage. Default: 0

    Definition Classes
    KnnModelParams
  74. final val outputFormat: Param[String]

    Permalink

    Param for the output format to produce.

    Param for the output format to produce. One of "full", "minimal" Setting this to minimal is more efficient when all you need is the identifier with its neighbors

    Default: "full"

    Definition Classes
    KnnModelParams
  75. final val parallelism: IntParam

    Permalink

    Param that specifies the number of threads to use.

    Param that specifies the number of threads to use. Default: number of processors available to the Java virtual machine

    Definition Classes
    KnnModelParams
  76. lazy val params: Array[Param[_]]

    Permalink
    Definition Classes
    Params
  77. final val partitionCol: Param[String]

    Permalink
    Definition Classes
    KnnAlgorithmParams
  78. final val predictionCol: Param[String]

    Permalink
    Definition Classes
    HasPredictionCol
  79. final val queryIdentifierCol: Param[String]

    Permalink

    Param for the column name for the query identifier.

    Param for the column name for the query identifier.

    Definition Classes
    KnnModelParams
  80. final val queryPartitionsCol: Param[String]

    Permalink

    Param for the column name for the query partitions.

    Param for the column name for the query partitions.

    Definition Classes
    KnnModelParams
  81. final def set(paramPair: ParamPair[_]): BruteForceSimilarity.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  82. final def set(param: String, value: Any): BruteForceSimilarity.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  83. final def set[T](param: Param[T], value: T): BruteForceSimilarity.this.type

    Permalink
    Definition Classes
    Params
  84. final def setDefault(paramPairs: ParamPair[_]*): BruteForceSimilarity.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  85. final def setDefault[T](param: Param[T], value: T): BruteForceSimilarity.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  86. def setDistanceFunction(value: String): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  87. def setExcludeSelf(value: Boolean): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  88. def setFeaturesCol(value: String): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  89. def setIdentifierCol(value: String): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  90. def setK(value: Int): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  91. def setNumPartitions(value: Int): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  92. def setNumReplicas(value: Int): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  93. def setOutputFormat(value: String): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  94. def setParallelism(value: Int): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  95. def setPartitionCol(value: String): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  96. def setPredictionCol(value: String): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  97. def setQueryIdentifierCol(value: String): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  98. def setQueryPartitionsCol(value: String): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  99. def setSimilarityThreshold(value: Double): BruteForceSimilarity.this.type

    Permalink

    Definition Classes
    KnnAlgorithm
  100. final val similarityThreshold: DoubleParam

    Permalink

    Param for the threshold value for inclusion.

    Param for the threshold value for inclusion. -1 indicates no threshold Default: -1

    Definition Classes
    KnnModelParams
  101. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  102. def toString(): String

    Permalink
    Definition Classes
    Identifiable → AnyRef → Any
  103. def transformSchema(schema: StructType): StructType

    Permalink
    Definition Classes
    KnnAlgorithm → PipelineStage
  104. def transformSchema(schema: StructType, logging: Boolean): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  105. val uid: String

    Permalink

    identifier

    identifier

    Definition Classes
    BruteForceSimilarity → KnnAlgorithm → Identifiable
  106. def validateAndTransformSchema(schema: StructType, identifierDataType: DataType): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    KnnModelParams
  107. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  108. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  109. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from KnnAlgorithm[BruteForceSimilarityModel]

Inherited from KnnAlgorithmParams

Inherited from KnnModelParams

Inherited from HasPredictionCol

Inherited from HasFeaturesCol

Inherited from Estimator[BruteForceSimilarityModel]

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

getParam

param

setParam

Ungrouped