gate.creole.tokeniser

Class DefaultTokeniser

java.lang.Object

gate.util.AbstractFeatureBearer

gate.creole.AbstractResource

gate.creole.AbstractProcessingResource

gate.creole.AbstractLanguageAnalyser

gate.creole.tokeniser.DefaultTokeniser

All Implemented Interfaces:

ANNIEConstants, Executable, LanguageAnalyser, ProcessingResource, Resource, Benchmarkable, FeatureBearer, NameBearer, Serializable

@CreoleResource(name="ANNIE English Tokeniser",
                comment="A customisable English tokeniser.",
                helpURL="http://gate.ac.uk/userguide/sec:annie:tokeniser",
                icon="tokeniser")
public class DefaultTokeniser
extends AbstractLanguageAnalyser
implements Benchmarkable

A composed tokeniser containing a SimpleTokeniser and a Transducer. The simple tokeniser tokenises the document and the transducer processes its output.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class gate.creole.AbstractProcessingResource

AbstractProcessingResource.InternalStatusListener, AbstractProcessingResource.IntervalProgressListener

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEF_TOK_ANNOT_SET_PARAMETER_NAME
static String	DEF_TOK_DOCUMENT_PARAMETER_NAME
static String	DEF_TOK_ENCODING_PARAMETER_NAME
static String	DEF_TOK_GRAMRULES_URL_PARAMETER_NAME
static String	DEF_TOK_TOKRULES_URL_PARAMETER_NAME
protected SimpleTokeniser	tokeniser the simple tokeniser used for tokenisation
protected Transducer	transducer the transducer used for post-processing

Fields inherited from class gate.creole.AbstractLanguageAnalyser

corpus, document

Fields inherited from class gate.creole.AbstractProcessingResource

interrupted

Fields inherited from class gate.creole.AbstractResource

name

Fields inherited from class gate.util.AbstractFeatureBearer

features

Fields inherited from interface gate.creole.ANNIEConstants

ANNOTATION_COREF_FEATURE_NAME, DATE_ANNOTATION_TYPE, DATE_POSTED_ANNOTATION_TYPE, DEFAULT_FILE, DOCUMENT_COREF_FEATURE_NAME, JOB_ID_ANNOTATION_TYPE, LOCATION_ANNOTATION_TYPE, LOOKUP_ANNOTATION_TYPE, LOOKUP_CLASS_FEATURE_NAME, LOOKUP_INSTANCE_FEATURE_NAME, LOOKUP_LANGUAGE_FEATURE_NAME, LOOKUP_MAJOR_TYPE_FEATURE_NAME, LOOKUP_MINOR_TYPE_FEATURE_NAME, LOOKUP_ONTOLOGY_FEATURE_NAME, MONEY_ANNOTATION_TYPE, ORGANIZATION_ANNOTATION_TYPE, PERSON_ANNOTATION_TYPE, PERSON_GENDER_FEATURE_NAME, PLUGIN_DIR, SENTENCE_ANNOTATION_TYPE, SPACE_TOKEN_ANNOTATION_TYPE, TOKEN_ANNOTATION_TYPE, TOKEN_CATEGORY_FEATURE_NAME, TOKEN_KIND_FEATURE_NAME, TOKEN_LENGTH_FEATURE_NAME, TOKEN_ORTH_FEATURE_NAME, TOKEN_STRING_FEATURE_NAME

Constructor Summary

Constructors

Constructor and Description
DefaultTokeniser()

Method Summary

Methods

Modifier and Type	Method and Description
void	cleanup() should clear all internal data of the resource.
void	execute() Run the resource.
String	getAnnotationSetName()
String	getBenchmarkId() Returns the benchmark ID of this resource.
String	getEncoding()
URL	getTokeniserRulesURL()
URL	getTransducerGrammarURL()
Resource	init() Initialise this resource, and return it.
void	interrupt() Notifies all the PRs in this controller that they should stop their execution as soon as possible.
void	setAnnotationSetName(String annotationSetName)
void	setBenchmarkId(String benchmarkId) This method sets the benchmarkID for this resource.
void	setEncoding(String encoding)
void	setTokeniserRulesURL(URL tokeniserRulesURL)
void	setTransducerGrammarURL(URL transducerGrammarURL)

Methods inherited from class gate.creole.AbstractLanguageAnalyser

getCorpus, getDocument, setCorpus, setDocument

Methods inherited from class gate.creole.AbstractProcessingResource

addProgressListener, addStatusListener, fireProcessFinished, fireProgressChanged, fireStatusChanged, getRuntimeParameterValues, getRuntimeParameterValues, isInterrupted, reInit, removeProgressListener, removeStatusListener

Methods inherited from class gate.creole.AbstractResource

checkParameterValues, flushBeanInfoCache, forgetBeanInfo, getBeanInfo, getInitParameterValues, getInitParameterValues, getName, getParameterValue, getParameterValue, getParameterValues, removeResourceListeners, setName, setParameterValue, setParameterValue, setParameterValues, setParameterValues, setResourceListeners, toString

Methods inherited from class gate.util.AbstractFeatureBearer

getFeatures, setFeatures

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface gate.ProcessingResource

reInit

Methods inherited from interface gate.Resource

getParameterValue, setParameterValue, setParameterValues

Methods inherited from interface gate.util.FeatureBearer

getFeatures, setFeatures

Methods inherited from interface gate.util.NameBearer

getName, setName

Methods inherited from interface gate.Executable

isInterrupted

Field Detail

DEF_TOK_DOCUMENT_PARAMETER_NAME

public static final String DEF_TOK_DOCUMENT_PARAMETER_NAME

See Also:

Constant Field Values

DEF_TOK_ANNOT_SET_PARAMETER_NAME

public static final String DEF_TOK_ANNOT_SET_PARAMETER_NAME

See Also:

Constant Field Values

DEF_TOK_TOKRULES_URL_PARAMETER_NAME

public static final String DEF_TOK_TOKRULES_URL_PARAMETER_NAME

See Also:

Constant Field Values

DEF_TOK_GRAMRULES_URL_PARAMETER_NAME

public static final String DEF_TOK_GRAMRULES_URL_PARAMETER_NAME

See Also:

Constant Field Values

DEF_TOK_ENCODING_PARAMETER_NAME

public static final String DEF_TOK_ENCODING_PARAMETER_NAME

See Also:

Constant Field Values

tokeniser

protected SimpleTokeniser tokeniser

the simple tokeniser used for tokenisation

transducer

protected Transducer transducer

the transducer used for post-processing

Constructor Detail

DefaultTokeniser

public DefaultTokeniser()

Method Detail

init

public Resource init()
              throws ResourceInstantiationException

Initialise this resource, and return it.

Specified by:

init in interface Resource

Overrides:

init in class AbstractProcessingResource

Throws:

ResourceInstantiationException

cleanup

public void cleanup()

Description copied from class: AbstractProcessingResource

should clear all internal data of the resource. Does nothing now

Specified by:

cleanup in interface Resource

Overrides:

cleanup in class AbstractProcessingResource

execute

public void execute()
             throws ExecutionException

Description copied from class: AbstractProcessingResource

Run the resource. It doesn't make sense not to override this in subclasses so the default implementation signals an exception.

Specified by:

execute in interface Executable

Overrides:

execute in class AbstractProcessingResource

Throws:

ExecutionException

interrupt

public void interrupt()

Notifies all the PRs in this controller that they should stop their execution as soon as possible.

Specified by:

interrupt in interface Executable

Overrides:

interrupt in class AbstractProcessingResource

setTokeniserRulesURL

@CreoleParameter(defaultValue="resources/tokeniser/DefaultTokeniser.rules",
                 comment="The URL to the rules file",
                 suffixes="rules")
public void setTokeniserRulesURL(URL tokeniserRulesURL)

getTokeniserRulesURL

public URL getTokeniserRulesURL()

setEncoding

@CreoleParameter(defaultValue="UTF-8",
                 comment="The encoding used for reading the definitions")
public void setEncoding(String encoding)

getEncoding

public String getEncoding()

setTransducerGrammarURL

@CreoleParameter(defaultValue="resources/tokeniser/postprocess.jape",
                 comment="The URL to the postprocessing transducer",
                 suffixes="jape")
public void setTransducerGrammarURL(URL transducerGrammarURL)

getTransducerGrammarURL

public URL getTransducerGrammarURL()

setAnnotationSetName

@RunTime
@Optional
@CreoleParameter(comment="The annotation set to be used for the generated annotations")
public void setAnnotationSetName(String annotationSetName)

getAnnotationSetName

public String getAnnotationSetName()

setBenchmarkId

public void setBenchmarkId(String benchmarkId)

Description copied from interface: Benchmarkable

This method sets the benchmarkID for this resource. The resource must use this as the prefix for any sub-events it logs.

Specified by:

setBenchmarkId in interface Benchmarkable

Parameters:

benchmarkId - the benchmark ID, which must not contain spaces as it is already used as a separator in the log, you can use Benchmark.createBenchmarkId(String, String) for it.

getBenchmarkId

public String getBenchmarkId()

Description copied from interface: Benchmarkable

Returns the benchmark ID of this resource.

Specified by:

getBenchmarkId in interface Benchmarkable