Class/Object

com.johnsnowlabs.nlp.annotators

Normalizer

Related Docs: object Normalizer | package annotators

Permalink

class Normalizer extends AnnotatorApproach[NormalizerModel]

Annotator that cleans out tokens. Requires stems, hence tokens. Removes all dirty characters from text following a regex pattern and transforms words based on a provided dictionary

For extended examples of usage, see the Spark NLP Workshop.

Example

import spark.implicits._
import com.johnsnowlabs.nlp.DocumentAssembler
import com.johnsnowlabs.nlp.annotator.{Normalizer, Tokenizer}
import org.apache.spark.ml.Pipeline
val documentAssembler = new DocumentAssembler()
  .setInputCol("text")
  .setOutputCol("document")

val tokenizer = new Tokenizer()
  .setInputCols("document")
  .setOutputCol("token")

val normalizer = new Normalizer()
  .setInputCols("token")
  .setOutputCol("normalized")
  .setLowercase(true)
  .setCleanupPatterns(Array("""[^\w\d\s]""")) // remove punctuations (keep alphanumeric chars)
// if we don't set CleanupPatterns, it will only keep alphabet letters ([^A-Za-z])

val pipeline = new Pipeline().setStages(Array(
  documentAssembler,
  tokenizer,
  normalizer
))

val data = Seq("John and Peter are brothers. However they don't support each other that much.")
  .toDF("text")
val result = pipeline.fit(data).transform(data)

result.selectExpr("normalized.result").show(truncate = false)
+----------------------------------------------------------------------------------------+
|result                                                                                  |
+----------------------------------------------------------------------------------------+
|[john, and, peter, are, brothers, however, they, dont, support, each, other, that, much]|
+----------------------------------------------------------------------------------------+
Linear Supertypes
AnnotatorApproach[NormalizerModel], CanBeLazy, DefaultParamsWritable, MLWritable, HasOutputAnnotatorType, HasOutputAnnotationCol, HasInputAnnotationCols, Estimator[NormalizerModel], PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By Inheritance
Inherited
  1. Normalizer
  2. AnnotatorApproach
  3. CanBeLazy
  4. DefaultParamsWritable
  5. MLWritable
  6. HasOutputAnnotatorType
  7. HasOutputAnnotationCol
  8. HasInputAnnotationCols
  9. Estimator
  10. PipelineStage
  11. Logging
  12. Params
  13. Serializable
  14. Serializable
  15. Identifiable
  16. AnyRef
  17. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Normalizer()

    Permalink

  2. new Normalizer(uid: String)

    Permalink

    uid

    required internal uid for saving annotator

Type Members

  1. type AnnotatorType = String

    Permalink
    Definition Classes
    HasOutputAnnotatorType

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def $[T](param: Param[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  4. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  5. def _fit(dataset: Dataset[_], recursiveStages: Option[PipelineModel]): NormalizerModel

    Permalink
    Attributes
    protected
    Definition Classes
    AnnotatorApproach

  6. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  7. def beforeTraining(spark: SparkSession): Unit

    Permalink
    Definition Classes
    AnnotatorApproach

  8. final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols

  9. val cleanupPatterns: StringArrayParam

    Permalink

    Normalization regex patterns which match will be removed from token (Default: Array("[^\\pL+]"))

  10. final def clear(param: Param[_]): Normalizer.this.type

    Permalink
    Definition Classes
    Params

  11. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  12. final def copy(extra: ParamMap): Estimator[NormalizerModel]

    Permalink
    Definition Classes
    AnnotatorApproach → Estimator → PipelineStage → Params

  13. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  14. final def defaultCopy[T <: Params](extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  15. val description: String

    Permalink

    Cleans out tokens

    Cleans out tokens

    Definition Classes
    Normalizer → AnnotatorApproach

  16. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  17. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  18. def explainParam(param: Param[_]): String

    Permalink
    Definition Classes
    Params

  19. def explainParams(): String

    Permalink
    Definition Classes
    Params

  20. final def extractParamMap(): ParamMap

    Permalink
    Definition Classes
    Params

  21. final def extractParamMap(extra: ParamMap): ParamMap

    Permalink
    Definition Classes
    Params

  22. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  23. final def fit(dataset: Dataset[_]): NormalizerModel

    Permalink
    Definition Classes
    AnnotatorApproach → Estimator

  24. def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): Seq[NormalizerModel]

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )

  25. def fit(dataset: Dataset[_], paramMap: ParamMap): NormalizerModel

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )

  26. def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): NormalizerModel

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" ) @varargs()

  27. final def get[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params

  28. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  29. def getCleanupPatterns: Array[String]

    Permalink

    Normalization regex patterns which match will be removed from token (Default: Array("[^\\pL+]"))

  30. final def getDefault[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params

  31. def getInputCols: Array[String]

    Permalink

    returns

    input annotations columns currently used

    Definition Classes
    HasInputAnnotationCols

  32. def getLazyAnnotator: Boolean

    Permalink
    Definition Classes
    CanBeLazy

  33. def getLowercase: Boolean

    Permalink

    Whether to convert strings to lowercase (Default: false)

  34. def getMaxLength: Int

    Permalink

    Set the maximum allowed length for each token

  35. def getMinLength: Int

    Permalink

    Set the minimum allowed length for each token (Default: 0)

  36. final def getOrDefault[T](param: Param[T]): T

    Permalink
    Definition Classes
    Params

  37. final def getOutputCol: String

    Permalink

    Gets annotation column name going to generate

    Gets annotation column name going to generate

    Definition Classes
    HasOutputAnnotationCol

  38. def getParam(paramName: String): Param[Any]

    Permalink
    Definition Classes
    Params

  39. def getSlangMatchCase: Boolean

    Permalink

    Whether or not to be case sensitive to match slangs (Default: false)

  40. final def hasDefault[T](param: Param[T]): Boolean

    Permalink
    Definition Classes
    Params

  41. def hasParam(paramName: String): Boolean

    Permalink
    Definition Classes
    Params

  42. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  43. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  44. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  45. val inputAnnotatorTypes: Array[String]

    Permalink

    Input Annotator Type : TOKEN

    Input Annotator Type : TOKEN

    Definition Classes
    Normalizer → HasInputAnnotationCols

  46. final val inputCols: StringArrayParam

    Permalink

    columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified

    columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified

    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols

  47. final def isDefined(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params

  48. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  49. final def isSet(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params

  50. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  51. val lazyAnnotator: BooleanParam

    Permalink
    Definition Classes
    CanBeLazy

  52. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  53. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  54. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  55. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  56. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  57. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  58. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  59. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  60. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  61. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  62. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  63. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  64. val lowercase: BooleanParam

    Permalink

    Whether to convert strings to lowercase (Default: false)

  65. val maxLength: IntParam

    Permalink

    Set the maximum allowed length for each token

  66. val minLength: IntParam

    Permalink

    Set the minimum allowed length for each token (Default: 0)

  67. def msgHelper(schema: StructType): String

    Permalink
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols

  68. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  69. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  70. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  71. def onTrained(model: NormalizerModel, spark: SparkSession): Unit

    Permalink
    Definition Classes
    AnnotatorApproach

  72. val optionalInputAnnotatorTypes: Array[String]

    Permalink
    Definition Classes
    HasInputAnnotationCols

  73. val outputAnnotatorType: AnnotatorType

    Permalink

    Output Annotator Type : TOKEN

    Output Annotator Type : TOKEN

    Definition Classes
    Normalizer → HasOutputAnnotatorType

  74. final val outputCol: Param[String]

    Permalink
    Attributes
    protected
    Definition Classes
    HasOutputAnnotationCol

  75. lazy val params: Array[Param[_]]

    Permalink
    Definition Classes
    Params

  76. def save(path: String): Unit

    Permalink
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )

  77. final def set(paramPair: ParamPair[_]): Normalizer.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  78. final def set(param: String, value: Any): Normalizer.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  79. final def set[T](param: Param[T], value: T): Normalizer.this.type

    Permalink
    Definition Classes
    Params

  80. def setCleanupPatterns(value: Array[String]): Normalizer.this.type

    Permalink

    Normalization regex patterns which match will be removed from token (Default: Array("[^\\pL+]"))

  81. final def setDefault(paramPairs: ParamPair[_]*): Normalizer.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  82. final def setDefault[T](param: Param[T], value: T): Normalizer.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  83. final def setInputCols(value: String*): Normalizer.this.type

    Permalink
    Definition Classes
    HasInputAnnotationCols

  84. def setInputCols(value: Array[String]): Normalizer.this.type

    Permalink

    Overrides required annotators column if different than default

    Overrides required annotators column if different than default

    Definition Classes
    HasInputAnnotationCols

  85. def setLazyAnnotator(value: Boolean): Normalizer.this.type

    Permalink
    Definition Classes
    CanBeLazy

  86. def setLowercase(value: Boolean): Normalizer.this.type

    Permalink

    Whether to convert strings to lowercase (Default: false)

  87. def setMaxLength(value: Int): Normalizer.this.type

    Permalink

    Set the maximum allowed length for each token

  88. def setMinLength(value: Int): Normalizer.this.type

    Permalink

    Set the minimum allowed length for each token (Default: 0)

  89. final def setOutputCol(value: String): Normalizer.this.type

    Permalink

    Overrides annotation column name when transforming

    Overrides annotation column name when transforming

    Definition Classes
    HasOutputAnnotationCol

  90. def setSlangDictionary(path: String, delimiter: String, readAs: Format = ReadAs.TEXT, options: Map[String, String] = Map("format" -> "text")): Normalizer.this.type

    Permalink

    Delimited file with list of custom words to be manually corrected

  91. def setSlangDictionary(value: ExternalResource): Normalizer.this.type

    Permalink

    Delimited file with list of custom words to be manually corrected

  92. def setSlangMatchCase(value: Boolean): Normalizer.this.type

    Permalink

    Whether or not to be case sensitive to match slangs (Default: false)

  93. val slangDictionary: ExternalResourceParam

    Permalink

    Delimited file with list of custom words to be manually corrected

  94. val slangMatchCase: BooleanParam

    Permalink

    Whether or not to be case sensitive to match slangs (Default: false)

  95. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  96. def toString(): String

    Permalink
    Definition Classes
    Identifiable → AnyRef → Any

  97. def train(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): NormalizerModel

    Permalink
    Definition Classes
    Normalizer → AnnotatorApproach

  98. final def transformSchema(schema: StructType): StructType

    Permalink

    requirement for pipeline transformation validation.

    requirement for pipeline transformation validation. It is called on fit()

    Definition Classes
    AnnotatorApproach → PipelineStage

  99. def transformSchema(schema: StructType, logging: Boolean): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()

  100. val uid: String

    Permalink

    required internal uid for saving annotator

    required internal uid for saving annotator

    Definition Classes
    Normalizer → Identifiable

  101. def validate(schema: StructType): Boolean

    Permalink

    takes a Dataset and checks to see if all the required annotation types are present.

    takes a Dataset and checks to see if all the required annotation types are present.

    schema

    to be validated

    returns

    True if all the required types are present, else false

    Attributes
    protected
    Definition Classes
    AnnotatorApproach

  102. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  103. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  104. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  105. def write: MLWriter

    Permalink
    Definition Classes
    DefaultParamsWritable → MLWritable

Inherited from AnnotatorApproach[NormalizerModel]

Inherited from CanBeLazy

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from HasOutputAnnotatorType

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from Estimator[NormalizerModel]

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.

Annotator types

Required input and expected output annotator types

Members

Parameter setters

Parameter getters