ws

Type Members

1. trait ReadablePretrainedWordSegmenter extends ParamsAndFeaturesReadable[WordSegmenterModel] with HasPretrained[WordSegmenterModel]

   

2. class WordSegmenterApproach extends AnnotatorApproach[WordSegmenterModel] with PerceptronTrainingUtils

   

   Trains a WordSegmenter which tokenizes non-english or non-whitespace separated texts.

   Trains a WordSegmenter which tokenizes non-english or non-whitespace separated texts.

   Many languages are not whitespace separated and their sentences are a concatenation of many symbols, like Korean, Japanese or Chinese. Without understanding the language, splitting the words into their corresponding tokens is impossible. The WordSegmenter is trained to understand these languages and split them into semantically correct parts.

   For instantiated/pretrained models, see WordSegmenterModel.

   To train your own model, a training dataset consisting of Part-Of-Speech tags is required. The data has to be loaded into a dataframe, where the column is an Annotation of type "POS". This can be set with setPosColumn.

   Tip: The helper class POS might be useful to read training data into data frames.

   For extended examples of usage, see the Spark NLP Workshop and the WordSegmenterTest.

   Example

   In this example, "chinese_train.utf8" is in the form of

   十|LL 四|RR 不|LL 是|RR 四|LL 十|RR

   and is loaded with the POS class to create a dataframe of "POS" type Annotations.

   import com.johnsnowlabs.nlp.base.DocumentAssembler
   import com.johnsnowlabs.nlp.annotators.ws.WordSegmenterApproach
   import com.johnsnowlabs.nlp.training.POS
   import org.apache.spark.ml.Pipeline
   
   val documentAssembler = new DocumentAssembler()
     .setInputCol("text")
     .setOutputCol("document")
   
   val wordSegmenter = new WordSegmenterApproach()
     .setInputCols("document")
     .setOutputCol("token")
     .setPosColumn("tags")
     .setNIterations(5)
   
   val pipeline = new Pipeline().setStages(Array(
     documentAssembler,
     wordSegmenter
   ))
   
   val trainingDataSet = POS().readDataset(
     spark,
     "src/test/resources/word-segmenter/chinese_train.utf8"
   )
   
   val pipelineModel = pipeline.fit(trainingDataSet)

3. class WordSegmenterModel extends AnnotatorModel[WordSegmenterModel] with HasSimpleAnnotate[WordSegmenterModel] with PerceptronPredictionUtils

   

   WordSegmenter which tokenizes non-english or non-whitespace separated texts.

   WordSegmenter which tokenizes non-english or non-whitespace separated texts.

   Many languages are not whitespace separated and their sentences are a concatenation of many symbols, like Korean, Japanese or Chinese. Without understanding the language, splitting the words into their corresponding tokens is impossible. The WordSegmenter is trained to understand these languages and plit them into semantically correct parts.

   This is the instantiated model of the WordSegmenterApproach. For training your own model, please see the documentation of that class.

   Pretrained models can be loaded with pretrained of the companion object:

   val wordSegmenter = WordSegmenterModel.pretrained()
     .setInputCols("document")
     .setOutputCol("words_segmented")

   The default model is "wordseg_pku", default language is "zh", if no values are provided. For available pretrained models please see the Models Hub.

   For extended examples of usage, see the Spark NLP Workshop and the WordSegmenterTest.

   Example

   import spark.implicits._
   import com.johnsnowlabs.nlp.base.DocumentAssembler
   import com.johnsnowlabs.nlp.annotator.WordSegmenterModel
   import org.apache.spark.ml.Pipeline
   
   val documentAssembler = new DocumentAssembler()
     .setInputCol("text")
     .setOutputCol("document")
   
   val wordSegmenter = WordSegmenterModel.pretrained()
     .setInputCols("document")
     .setOutputCol("token")
   
   val pipeline = new Pipeline().setStages(Array(
     documentAssembler,
     wordSegmenter
   ))
   
   val data = Seq("然而，這樣的處理也衍生了一些問題。").toDF("text")
   val result = pipeline.fit(data).transform(data)
   
   result.select("token.result").show(false)
   +--------------------------------------------------------+
   |result                                                  |
   +--------------------------------------------------------+
   |[然而, ，, 這樣, 的, 處理, 也, 衍生, 了, 一些, 問題, 。    ]|
   +--------------------------------------------------------+