Class

org.apache.spark.ml.clustering.tupol

XKMeans

Related Doc: package tupol

Permalink

class XKMeans extends KMeans with XKMeansParams

Extended KMeans algorithm.

Calculates the following: - cluster (prediction); already available in the default KMeans algorithm. - distance to cluster - probability - probability by feature (dimension)

Note: The probability by feature algorithm is based on the ideas presented in https://github.com/tupol/naive-ml; https://github.com/tupol/naive-ml/blob/master/src/main/scala/tupol/ml/clustering/KMeansGaussian.scala.

Note: The probability by feature algorithm can be rendered useless if a feature/dimension reduction algorithm is used before applying XKMeans2, as we will be unable to track back the exact feature which contributed to a record being classified as an anomaly.

Note: This is by far not a perfect solution yet, as the general assumption is that the data follows a normal distribution, which is not always the case.

Annotations
@Experimental()
Linear Supertypes
XKMeansParams, HasProbabilityByFeatureCol, HasProbabilityCol, HasDistanceToCentroidCol, KMeans, DefaultParamsWritable, MLWritable, KMeansParams, HasDistanceMeasure, HasTol, HasPredictionCol, HasSeed, HasFeaturesCol, HasMaxIter, Estimator[KMeansModel], PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. XKMeans
  2. XKMeansParams
  3. HasProbabilityByFeatureCol
  4. HasProbabilityCol
  5. HasDistanceToCentroidCol
  6. KMeans
  7. DefaultParamsWritable
  8. MLWritable
  9. KMeansParams
  10. HasDistanceMeasure
  11. HasTol
  12. HasPredictionCol
  13. HasSeed
  14. HasFeaturesCol
  15. HasMaxIter
  16. Estimator
  17. PipelineStage
  18. Logging
  19. Params
  20. Serializable
  21. Serializable
  22. Identifiable
  23. AnyRef
  24. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new XKMeans()

    Permalink
  2. new XKMeans(uid: String)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  4. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  6. final def clear(param: Param[_]): XKMeans.this.type

    Permalink
    Definition Classes
    Params
  7. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  8. def copy(extra: ParamMap): KMeans

    Permalink
    Definition Classes
    KMeans → Estimator → PipelineStage → Params
    Annotations
    @Since( "1.5.0" )
  9. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  10. final def defaultCopy[T <: Params](extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  11. final val distanceMeasure: Param[String]

    Permalink
    Definition Classes
    HasDistanceMeasure
  12. final val distanceToCentroidCol: Param[String]

    Permalink

    Param for distanceToCentroid column name.

    Param for distanceToCentroid column name.

    Definition Classes
    HasDistanceToCentroidCol
  13. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  14. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  15. def explainParam(param: Param[_]): String

    Permalink
    Definition Classes
    Params
  16. def explainParams(): String

    Permalink
    Definition Classes
    Params
  17. final def extractParamMap(): ParamMap

    Permalink
    Definition Classes
    Params
  18. final def extractParamMap(extra: ParamMap): ParamMap

    Permalink
    Definition Classes
    Params
  19. final val featureNames: Param[Option[Seq[String]]]

    Permalink

    Set the names of the features corresponding to each dimension of the input vectors column

    Set the names of the features corresponding to each dimension of the input vectors column

    Definition Classes
    XKMeansParams
  20. final val featuresCol: Param[String]

    Permalink
    Definition Classes
    HasFeaturesCol
  21. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  22. def fit(dataset: Dataset[_]): XKMeansModel

    Permalink
    Definition Classes
    XKMeans → KMeans → Estimator
  23. def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): Seq[KMeansModel]

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  24. def fit(dataset: Dataset[_], paramMap: ParamMap): KMeansModel

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  25. def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): KMeansModel

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" ) @varargs()
  26. final def get[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  27. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  28. final def getDefault[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  29. final def getDistanceMeasure: String

    Permalink
    Definition Classes
    HasDistanceMeasure
  30. final def getDistanceToCentroidCol: String

    Permalink

    Definition Classes
    HasDistanceToCentroidCol
  31. def getFeatureNames: Option[Seq[String]]

    Permalink

    Definition Classes
    XKMeansParams
  32. final def getFeaturesCol: String

    Permalink
    Definition Classes
    HasFeaturesCol
  33. def getInitMode: String

    Permalink
    Definition Classes
    KMeansParams
    Annotations
    @Since( "1.5.0" )
  34. def getInitSteps: Int

    Permalink
    Definition Classes
    KMeansParams
    Annotations
    @Since( "1.5.0" )
  35. def getK: Int

    Permalink
    Definition Classes
    KMeansParams
    Annotations
    @Since( "1.5.0" )
  36. final def getMaxIter: Int

    Permalink
    Definition Classes
    HasMaxIter
  37. final def getOrDefault[T](param: Param[T]): T

    Permalink
    Definition Classes
    Params
  38. def getParam(paramName: String): Param[Any]

    Permalink
    Definition Classes
    Params
  39. final def getPredictionCol: String

    Permalink
    Definition Classes
    HasPredictionCol
  40. final def getProbabilityByFeatureCol: String

    Permalink

    Definition Classes
    HasProbabilityByFeatureCol
  41. final def getProbabilityCol: String

    Permalink
    Definition Classes
    HasProbabilityCol
  42. final def getSeed: Long

    Permalink
    Definition Classes
    HasSeed
  43. final def getTol: Double

    Permalink
    Definition Classes
    HasTol
  44. def getXSigma: Double

    Permalink

    Definition Classes
    XKMeansParams
  45. final def hasDefault[T](param: Param[T]): Boolean

    Permalink
    Definition Classes
    Params
  46. def hasParam(paramName: String): Boolean

    Permalink
    Definition Classes
    Params
  47. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  48. final val initMode: Param[String]

    Permalink
    Definition Classes
    KMeansParams
    Annotations
    @Since( "1.5.0" )
  49. final val initSteps: IntParam

    Permalink
    Definition Classes
    KMeansParams
    Annotations
    @Since( "1.5.0" )
  50. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  51. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  52. final def isDefined(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  53. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  54. final def isSet(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  55. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  56. final val k: IntParam

    Permalink
    Definition Classes
    KMeansParams
    Annotations
    @Since( "1.5.0" )
  57. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  58. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  59. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  60. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  61. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  62. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  63. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  64. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  65. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  66. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  67. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  68. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  69. final val maxIter: IntParam

    Permalink
    Definition Classes
    HasMaxIter
  70. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  71. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  72. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  73. lazy val params: Array[Param[_]]

    Permalink
    Definition Classes
    Params
  74. final val predictionCol: Param[String]

    Permalink
    Definition Classes
    HasPredictionCol
  75. final val probabilityByFeatureCol: Param[String]

    Permalink

    Param for probabilityByFeature column name.

    Param for probabilityByFeature column name.

    Definition Classes
    HasProbabilityByFeatureCol
  76. final val probabilityCol: Param[String]

    Permalink
    Definition Classes
    HasProbabilityCol
  77. def save(path: String): Unit

    Permalink
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )
  78. final val seed: LongParam

    Permalink
    Definition Classes
    HasSeed
  79. final def set(paramPair: ParamPair[_]): XKMeans.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  80. final def set(param: String, value: Any): XKMeans.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  81. final def set[T](param: Param[T], value: T): XKMeans.this.type

    Permalink
    Definition Classes
    Params
  82. final def setDefault(paramPairs: ParamPair[_]*): XKMeans.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  83. final def setDefault[T](param: Param[T], value: T): XKMeans.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  84. def setDistanceMeasure(value: String): XKMeans.this.type

    Permalink
    Definition Classes
    KMeans
    Annotations
    @Since( "2.4.0" )
  85. def setFeatureNames(value: Option[Seq[String]]): XKMeans.this.type

    Permalink
  86. def setFeaturesCol(value: String): XKMeans.this.type

    Permalink
    Definition Classes
    KMeans
    Annotations
    @Since( "1.5.0" )
  87. def setInitMode(value: String): XKMeans.this.type

    Permalink
    Definition Classes
    KMeans
    Annotations
    @Since( "1.5.0" )
  88. def setInitSteps(value: Int): XKMeans.this.type

    Permalink
    Definition Classes
    KMeans
    Annotations
    @Since( "1.5.0" )
  89. def setK(value: Int): XKMeans.this.type

    Permalink
    Definition Classes
    KMeans
    Annotations
    @Since( "1.5.0" )
  90. def setMaxIter(value: Int): XKMeans.this.type

    Permalink
    Definition Classes
    KMeans
    Annotations
    @Since( "1.5.0" )
  91. def setPredictionCol(value: String): XKMeans.this.type

    Permalink
    Definition Classes
    KMeans
    Annotations
    @Since( "1.5.0" )
  92. def setSeed(value: Long): XKMeans.this.type

    Permalink
    Definition Classes
    KMeans
    Annotations
    @Since( "1.5.0" )
  93. def setTol(value: Double): XKMeans.this.type

    Permalink
    Definition Classes
    KMeans
    Annotations
    @Since( "1.5.0" )
  94. def setXSigma(value: Double): XKMeans.this.type

    Permalink
    Definition Classes
    XKMeansParams
  95. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  96. def toString(): String

    Permalink
    Definition Classes
    Identifiable → AnyRef → Any
  97. final val tol: DoubleParam

    Permalink
    Definition Classes
    HasTol
  98. def transformSchema(schema: StructType): StructType

    Permalink
    Definition Classes
    KMeans → PipelineStage
    Annotations
    @Since( "1.5.0" )
  99. def transformSchema(schema: StructType, logging: Boolean): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  100. val uid: String

    Permalink
    Definition Classes
    XKMeans → KMeans → Identifiable
  101. def validateAndTransformSchema(schema: StructType): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    KMeansParams
  102. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  103. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  104. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  105. def write: MLWriter

    Permalink
    Definition Classes
    DefaultParamsWritable → MLWritable
  106. final val xSigma: DoubleParam

    Permalink

    The ratio of standard deviation used to compute probability

    The ratio of standard deviation used to compute probability

    Definition Classes
    XKMeansParams

Inherited from XKMeansParams

Inherited from HasProbabilityByFeatureCol

Inherited from HasProbabilityCol

Inherited from HasDistanceToCentroidCol

Inherited from KMeans

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from KMeansParams

Inherited from HasDistanceMeasure

Inherited from HasTol

Inherited from HasPredictionCol

Inherited from HasSeed

Inherited from HasFeaturesCol

Inherited from HasMaxIter

Inherited from Estimator[KMeansModel]

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

getParam

param

Ungrouped