Trains a WordSegmenter which tokenizes non-english or non-whitespace separated texts.
WordSegmenter which tokenizes non-english or non-whitespace separated texts.
WordSegmenter which tokenizes non-english or non-whitespace separated texts.
Many languages are not whitespace separated and their sentences are a concatenation of many symbols, like Korean, Japanese or Chinese. Without understanding the language, splitting the words into their corresponding tokens is impossible. The WordSegmenter is trained to understand these languages and plit them into semantically correct parts.
This is the instantiated model of the WordSegmenterApproach. For training your own model, please see the documentation of that class.
Pretrained models can be loaded with pretrained
of the companion object:
val wordSegmenter = WordSegmenterModel.pretrained() .setInputCols("document") .setOutputCol("words_segmented")
The default model is "wordseg_pku"
, default language is "zh"
, if no values are provided.
For available pretrained models please see the Models Hub.
For extended examples of usage, see the Spark NLP Workshop and the WordSegmenterTest.
import spark.implicits._ import com.johnsnowlabs.nlp.base.DocumentAssembler import com.johnsnowlabs.nlp.annotator.WordSegmenterModel import org.apache.spark.ml.Pipeline val documentAssembler = new DocumentAssembler() .setInputCol("text") .setOutputCol("document") val wordSegmenter = WordSegmenterModel.pretrained() .setInputCols("document") .setOutputCol("token") val pipeline = new Pipeline().setStages(Array( documentAssembler, wordSegmenter )) val data = Seq("然而,這樣的處理也衍生了一些問題。").toDF("text") val result = pipeline.fit(data).transform(data) result.select("token.result").show(false) +--------------------------------------------------------+ |result | +--------------------------------------------------------+ |[然而, ,, 這樣, 的, 處理, 也, 衍生, 了, 一些, 問題, 。 ]| +--------------------------------------------------------+
This is the companion object of WordSegmenterApproach.
This is the companion object of WordSegmenterApproach. Please refer to that class for the documentation.
This is the companion object of WordSegmenterModel.
This is the companion object of WordSegmenterModel. Please refer to that class for the documentation.
Trains a WordSegmenter which tokenizes non-english or non-whitespace separated texts.
Many languages are not whitespace separated and their sentences are a concatenation of many symbols, like Korean, Japanese or Chinese. Without understanding the language, splitting the words into their corresponding tokens is impossible. The WordSegmenter is trained to understand these languages and split them into semantically correct parts.
For instantiated/pretrained models, see WordSegmenterModel.
To train your own model, a training dataset consisting of Part-Of-Speech tags is required. The data has to be loaded into a dataframe, where the column is an Annotation of type
"POS"
. This can be set withsetPosColumn
.Tip: The helper class POS might be useful to read training data into data frames.
For extended examples of usage, see the Spark NLP Workshop and the WordSegmenterTest.
Example
In this example,
"chinese_train.utf8"
is in the form ofand is loaded with the
POS
class to create a dataframe of"POS"
type Annotations.