com.johnsnowlabs.nlp.annotators
DocumentNormalizer
Related Docs:
object DocumentNormalizer
| package annotators
com.johnsnowlabs.nlp.annotators
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
action to perform applying regex patterns on text
takes a document and annotations and produces new annotations of this annotator's annotation type
takes a document and annotations and produces new annotations of this annotator's annotation type
Annotations that correspond to inputAnnotationCols generated by previous annotators if any
any number of annotations processed for every input annotation. Not necessary one to one relationship
requirement for annotators copies
requirement for annotators copies
Wraps annotate to happen inside SparkSQL user defined functions in order to act with org.apache.spark.sql.Column
Wraps annotate to happen inside SparkSQL user defined functions in order to act with org.apache.spark.sql.Column
udf function to be applied to inputCols using this annotator's annotate function as part of ML transformation
file encoding to apply on normalized documents
Override for additional custom schema checks
Override for additional custom schema checks
Action to perform on text.
Action to perform on text. Default "clean".
Encoding to apply to normalized documents.
input annotations columns currently used
Lowercase tokens, default false
Gets annotation column name going to generate
Gets annotation column name going to generate
Regular expressions list for normalization.
Policy to remove patterns from text.
Policy to remove patterns from text. Defaults "pretty_all".
Replacement string to apply when regexes match
Input annotator type : DOCUMENT
Input annotator type : DOCUMENT
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
whether to convert strings to lowercase
Input annotator type : DOCUMENT
Input annotator type : DOCUMENT
normalization regex patterns which match will be removed from document
removalPolicy to remove patterns from text with a given policy
replacement string to apply when regexes match
Action to perform on text.
Action to perform on text. Default "clean".
Encoding to apply.
Encoding to apply. Default is UTF-8. Valid encoding are values are: UTF_8, UTF_16, US_ASCII, ISO-8859-1, UTF-16BE, UTF-16LE
Overrides required annotators column if different than default
Overrides required annotators column if different than default
Lower case tokens, default false
Overrides annotation column name when transforming
Overrides annotation column name when transforming
Regular expressions list for normalization.
Removal policy to apply.
Removal policy to apply. Valid policy values are: "all", "pretty_all", "first", "pretty_first"
Replacement string to apply when regexes match
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
Dataset[Row]
requirement for pipeline transformation validation.
requirement for pipeline transformation validation. It is called on fit()
required uid for storing annotator to disk
required uid for storing annotator to disk
takes a Dataset and checks to see if all the required annotation types are present.
takes a Dataset and checks to see if all the required annotation types are present.
to be validated
True if all the required types are present, else false
A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.
Required input and expected output annotator types
Annotator which normalizes raw text from tagged text, e.g. scraped web pages or xml documents, from document type columns into Sentence. Removes all dirty characters from text following one or more input regex patterns. Can apply not wanted character removal with a specific policy. Can apply lower case normalization.
See DocumentNormalizer test class for examples examples of usage.