Class | Description |
---|---|
AmbiguityClass |
An ambiguity class for a word is the word by itself or its set of observed tags.
|
AmbiguityClasses |
A collection of Ambiguity Class.
|
ASBCunkDict | |
CountWrapper |
A simple data structure for some tag counts.
|
CtbDict | |
CTBunkDict | |
DataWordTag | |
Dictionary |
Maintains a map from words to tags and their counts.
|
DictionaryExtractor |
This class is the same as a regular Extractor, but keeps a pointer
to the tagger's dictionary as well.
|
Distsim |
Keeps track of a distributional similarity mapping, i.e., a map from
word to class.
|
Extractor |
This class serves as the base class for classes which extract relevant
information from a history to give it to the features.
|
ExtractorDistsim |
Extractor for adding distsim information.
|
ExtractorDistsimConjunction |
Extractor for adding a conjunction of distsim information.
|
ExtractorFrames |
This class contains the basic feature extractors used for all words and
tag sequences (and interaction terms) for the MaxentTagger, but not the
feature extractors explicitly targeting generalization for rare or unknown
words.
|
ExtractorFramesRare |
This class contains feature extractors for the MaxentTagger that are only
applied to rare (low frequency/unknown) words.
|
Extractors |
Maintains a set of feature extractors for a maxent POS tagger and applies them.
|
ExtractorVerbalVBNZero |
Look for verbs selecting a VBN verb.
|
FeatureKey |
Stores a triple of an extractor ID, a feature value (derived from history)
and a y (tag) value.
|
History |
A History is used to look inside a PairsHolder with indices for the start and end of a sentence and
the current position.
|
HistoryTable |
This maintains a two way lookup between a History and
an Integer index.
|
LambdaSolveTagger |
This module does the working out of lambda parameters for binary tagger
features.
|
MaxentTagger |
The main class for users to run, train, and test the part of speech tagger.
|
MaxentTaggerGUI |
A very simple GUI for illustrating the POS tagger tagging text.
|
MaxentTaggerServer |
A POS tagger server for the Stanford POS Tagger.
|
PairsHolder |
A simple class that maintains a list of WordTag pairs which are interned
as they are added.
|
ReadDataTagged |
Reads tagged data from a file and creates a dictionary.
|
TaggerConfig |
Reads and stores configuration information for a POS tagger.
|
TaggerExperiments |
This class represents the training samples.
|
TaggerFeature |
Holds a Tagger Feature for the loglinear model.
|
TaggerFeatures |
This class contains POS tagger specific features.
|
TemplateHash | |
TestClassifier |
Tags data and can handle either data with gold-standard tags (computing
performance statistics) or unlabeled data.
|
TestSentence | |
TTags |
This class holds the POS tags, assigns them unique ids, and knows which tags
are open versus closed class.
|
Enum | Description |
---|---|
TaggerConfig.Mode |
A Maximum Entropy Part-of-Speech Tagger. It can run either a Conditional Markov Model (CMM) aka Maximum Entropy Markov Model (MEMM) tagger or a cyclic dependency network tagger.
If you are only interested in using one of the trained taggers
included in the distribution, either from the commandline or via the
Java API, look at the documentation of the class
MaxentTagger
.
Or, if you are interested in training a tagger from data using
some of the built-in architecture options (CMM or bi-directional
dependency network), also look at the documentation for
MaxentTagger
.
The rest of this document is for more complex situations where you want to define the features/architecture of your own tagger, which requires delving into the code.
The pre-defined features are for CMMs and bi-directional dependency networks. The local models are log-linear models using features specified via feature templates.
There are two kinds of templates: ones for rare words, and ones for common words. For a context centered at a common word, only common word features are active. For a context centered at a rare word, both common and rare word features are active. Which words are considered common and which rare is determined by the parameter GlobalHolder.rareThreshold. For example, a threshold of five means that words occurring five times or less are considered rare.
The feature templates represent conditions on the words and tags
surrounding the current position and also the target tag. For example
<t_-1,t_0>
is a common word
feature template. It is
instantiated for various values of the previous and current tag. A feature
formed by instantiating this template will look for example like this:
<t_-1=DT,t_0=NN>
and will have value 1 for a history h and tag t
iff the condition is satisfied. Every feature
template includes a specification of the current tag. It is not possible at the
moment to include features that are true if the current tag is one of several
possible, e.g. NN or NNS or NNP.
To reduce the number
of features, cutoffs on the number of times a feature is active are introduced.
The cutoff for common word features is GlobalHolder.threshold, and
the cutoff for rare word features is GlobalHolder.thresholdRare.
The thresholds work like this: the part of the feature that does not include
the tag has to be active in the training set at least cutoff+1 times, and the
complete feature has to be active at least once in order for the feature to be
included in the model. (Note, the cutoff for the current word feature is set to
2 independent of threshold settings; cf. method
TaggerExperiments.populated(int, int)
.
In order to train a tagger, we need to specify the feature templates to be used, change the count cutoffs if we want, change the default parameter estimation method if we want, perhaps hand-specify closed class POS tags, and then train given tagged text.
Feature templates inherit from the class Extractor
.
The main job of an Extractor is to extract the value it is interested in from a history.
Each instantiating feature for a given template will be true for a specific
value extracted from a history and a specific target tag.
For example, this is a common word extractor that extracts the current and next word.
/** * This extractor extracts the current and the next word in conjunction. *\/ class ExtractorCWordNextWord extends Extractor { private final static String excl="!"; public ExtractorCWordNextWord() {} String extract(History h, PairsHolder pH) { String s = pH.get(h, 0, false) + excl + pH.get(h, 1, false); return s; } }
The method extract(History h) is defined in the base class Extractor as:
*String extract(History h) { return extract(h, GlobalHolder.pairs); }
The PairsHolder contains an array of words and the tags. It has a get method that can be used to extract things from the history. In GlobalHolder.pairs , the whole training data is stored. String GlobalHolder.pairs.get(History h,int position, boolean isTag) , will return the tag or word (depending on isTag), at position position relative to the history h.
Using this PairsHolder, we can extract features from the whole sentence including the current word. The History object is basically a specification of the start of the sentence, the current word, and the end of the sentence.
In an extractor we can also specify for which tags to
instantiate the template. The method
boolean precondition(String tag)
is by default true, meaning
that a feature can be created for every tag. Sometimes we would like to
restrict that, and say that features should be created for only the VB and VBP
tags, for example. In this case the method precondition has to be redefined to
return false for all other tags.
The extractors for common word features have to be placed in the static array ExtractorFrames.eFrames. The present state of this array for the best tagger is:
public static Extractor[] eFrames={cWord,prevWord,nextWord,prevTag,nextTag, prevTwoTags,nextTwoTags,prevNextTag,prevTagWord,nextTagWord,cWordPrevWord, cWordNextWord};
The extractors for rare word features commonly inherit from RareExtractor, which inherits from Extractor. RareExtractor provides some nice static methods for manipulating strings, such as seeing whether they contain numbers, etc. The rare word extractors have to be placed in the static array ExtractorFramesRare.eFrames. For example, for a good English tagger, this array might be:
public static Extractor[] eFrames={cWordUppCase,cWordNumber, cWordDash,cWordSuff1,cWordSuff2,cWordSuff3,cWordSuff4, cAllCap,cMidSentence,cWordStartUCase,cWordMidUCase, cWordPref1,cWordPref2,cWordPref3,cWordPref4, new ExtractorCWordPref(5),new ExtractorCWordPref(6), new ExtractorCWordPref(7), new ExtractorCWordPref(8), new ExtractorCWordPref(9), new ExtractorCWordPref(10), new ExtractorCWordSuff(5),new ExtractorCWordSuff(6), new ExtractorCWordSuff(7),new ExtractorCWordSuff(8), new ExtractorCWordSuff(9), new ExtractorCWordSuff(10), cLetterDigitDash, cCompany,cAllCapitalized,cUpperDigitDash};
At present, many of the extractor and rare extractor combinations can
be flexibly set from a properties file by suitable specifications of the
arch
option, whereas others require changing the code.
By default, all POS tags are assumed to be open classes. In many cases,
it is useful to specify POS tags which are closed class, and can only be
applied to words seen in the training data (rather than being possible
tags for new words seen at runtime). These closed class tags are
specified for a language (where a "language" is really a
(language,tag-set) pair: a different system of tagging is a new
language). You do this by specifying the language in the properties
file, and specifying the closed class tags for that language in
TTags
.