breeze.text.tokenize
PTBTokenizer tokenizes sentences into treebank style sentences. Input must be a single sentence.
Should be replaced by a real lexer one day, meh.
PTBTokenizer tokenizes sentences into treebank style sentences. Input must be a single sentence.
Should be replaced by a real lexer one day, meh.