com.johnsnowlabs.nlp.annotators.seq2seq
T5Transformer
Related Docs:
object T5Transformer
| package seq2seq
com.johnsnowlabs.nlp.annotators.seq2seq
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
takes a document and annotations and produces new annotations of this annotator's annotation type
takes a document and annotations and produces new annotations of this annotator's annotation type
Annotations in batches that correspond to inputAnnotationCols generated by previous annotators if any
any number of annotations processed for every batch of input annotations. Not necessary one to one relationship IMPORTANT: !MUST! return sequences of equal lengths !! IMPORTANT: !MUST! return sentences that belong to the same original row !! (challenging)
Size of every batch (Default depends on model).
Size of every batch (Default depends on model).
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()
requirement for annotators copies
requirement for annotators copies
Whether or not to use sampling, use greedy decoding otherwise (Default: false)
Override for additional custom schema checks
Override for additional custom schema checks
Size of every batch.
Size of every batch.
input annotations columns currently used
Gets annotation column name going to generate
Gets annotation column name going to generate
A list of token ids which are ignored in the decoder's output
Input annotator type : DOCUMENT
Input annotator type : DOCUMENT
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
Maximum length of the sequence to be generated (Default: 20)
Minimum length of the sequence to be generated (Default: 0)
If set to int > 0, all ngrams of that size can only occur once (Default: 0)
Output annotator type : DOCUMENT
Output annotator type : DOCUMENT
Optional Random seed for the model.
Optional Random seed for the model. Needs to be of type Long.
The parameter for repetition penalty (Default: 1.0).
The parameter for repetition penalty (Default: 1.0). 1.0 means no penalty. See
this paper for more details.
Size of every batch.
Size of every batch.
Overrides required annotators column if different than default
Overrides required annotators column if different than default
Overrides annotation column name when transforming
Overrides annotation column name when transforming
Set transformer task, e.g.
Set transformer task, e.g. "summarize:" (Default: ""). The T5 task needs to be in the
format "task:".
The value used to module the next token probabilities (Default: 1.0)
The number of highest probability vocabulary tokens to keep for top-k-filtering (Default:
50)
If set to float < 1.0, only the most probable tokens with probabilities that add up to
topP or higher are kept for generation (Default: 1.0)
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
Dataset[Row]
requirement for pipeline transformation validation.
requirement for pipeline transformation validation. It is called on fit()
required uid for storing annotator to disk
required uid for storing annotator to disk
takes a Dataset and checks to see if all the required annotation types are present.
takes a Dataset and checks to see if all the required annotation types are present.
to be validated
True if all the required types are present, else false
A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.
T5: the Text-To-Text Transfer Transformer
T5 reconsiders all NLP tasks into a unified text-to-text-format where the input and output are always text strings, in contrast to BERT-style models that can only output either a class label or a span of the input. The text-to-text framework is able to use the same model, loss function, and hyper-parameters on any NLP task, including machine translation, document summarization, question answering, and classification tasks (e.g., sentiment analysis). T5 can even apply to regression tasks by training it to predict the string representation of a number instead of the number itself.
Pretrained models can be loaded with
pretrainedof the companion object:The default model is
"t5_small", if no name is provided. For available pretrained models please see the Models Hub.For extended examples of usage, see the Spark NLP Workshop and the T5TestSpec.
Sources:
Paper Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new Colossal Clean Crawled Corpus, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.
Note:
This is a very computationally expensive module especially on larger sequence. The use of an accelerator such as GPU is recommended.
Example