edu.stanford.nlp.parser.lexparser

Class ChineseMaxentLexicon

java.lang.Object

edu.stanford.nlp.parser.lexparser.ChineseMaxentLexicon

All Implemented Interfaces:

Lexicon, java.io.Serializable

public class ChineseMaxentLexicon
extends java.lang.Object
implements Lexicon

A Lexicon class that computes the score of word|tag according to a maxent model of tag|word (divided by MLE estimate of P(tag)).
It would be nice to factor out a superclass MaxentLexicon that takes a WordFeatureExtractor

Author:

Galen Andrew

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static boolean	fixUnkFunctionWords
static boolean	seenTagsOnly
CollectionValuedMap<java.lang.String,java.lang.String>	tagsForWord

Fields inherited from interface edu.stanford.nlp.parser.lexparser.Lexicon

BOUNDARY, BOUNDARY_TAG, UNKNOWN_WORD

Constructor Summary

Constructors

Constructor and Description
ChineseMaxentLexicon(Options op, Index<java.lang.String> wordIndex, Index<java.lang.String> tagIndex, int featureLevel)

Method Summary

Modifier and Type	Method and Description
void	finishTraining() Done collecting statistics for the lexicon.
UnknownWordModel	getUnknownWordModel()
void	incrementTreesRead(double weight) If training on a per-word basis instead of on a per-tree basis, we will want to increment the tree count as this happens.
void	initializeTraining(double numTrees) Start training this lexicon on the expected number of trees.
boolean	isKnown(int word) Checks whether a word is in the lexicon.
boolean	isKnown(java.lang.String word) Checks whether a word is in the lexicon.
static void	main(java.lang.String[] args)
int	numRules() Returns the number of rules (tag rewrites as word) in the Lexicon.
void	readData(java.io.BufferedReader in) Read the lexicon from the BufferedReader in the format written by writeData.
java.util.Iterator<IntTaggedWord>	ruleIteratorByWord(int word, int loc, java.lang.String featureSpec) Get an iterator over all rules (pairs of (word, POS)) for this word.
java.util.Iterator<IntTaggedWord>	ruleIteratorByWord(java.lang.String word, int loc, java.lang.String featureSpec) Same thing, but with a string that needs to be translated by the lexicon's word index
float	score(IntTaggedWord iTW, int loc, java.lang.String word, java.lang.String featureSpec) Get the score of this word with this tag (as an IntTaggedWord) at this loc.
void	setUnknownWordModel(UnknownWordModel uwm)
java.util.Set<java.lang.String>	tagSet(java.util.function.Function<java.lang.String,java.lang.String> basicCategoryFunction) Return the Set of tags used by this tagger (available after training the tagger).
void	train(java.util.Collection<Tree> trees) Add the given collection of trees to the statistics counted.
void	train(java.util.Collection<Tree> trees, java.util.Collection<Tree> rawTrees)
void	train(java.util.Collection<Tree> trees, double weight) Add the given collection of trees to the statistics counted.
void	train(java.util.List<TaggedWord> sentence, double weight) Add the given sentence to the statistics counted.
void	train(TaggedWord tw, int loc, double weight) Not all subclasses support this particular method.
void	train(Tree tree, double weight) Add the given tree to the statistics counted.
void	trainUnannotated(java.util.List<TaggedWord> sentence, double weight) Sometimes we might have a sentence of tagged words which we would like to add to the lexicon, but they weren't part of a binarized, markovized, or otherwise annotated tree.
void	writeData(java.io.Writer w) Write the lexicon in human-readable format to the Writer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

seenTagsOnly

public static final boolean seenTagsOnly

See Also:

Constant Field Values

fixUnkFunctionWords

public static final boolean fixUnkFunctionWords

See Also:

Constant Field Values

tagsForWord

public CollectionValuedMap<java.lang.String,java.lang.String> tagsForWord

Constructor Detail

ChineseMaxentLexicon

public ChineseMaxentLexicon(Options op,
                            Index<java.lang.String> wordIndex,
                            Index<java.lang.String> tagIndex,
                            int featureLevel)

Method Detail

isKnown

public boolean isKnown(int word)

Description copied from interface: Lexicon

Checks whether a word is in the lexicon.

Specified by:

isKnown in interface Lexicon

Parameters:

word - The word as an int

Returns:

Whether the word is in the lexicon

isKnown

public boolean isKnown(java.lang.String word)

Description copied from interface: Lexicon

Checks whether a word is in the lexicon.

Specified by:

isKnown in interface Lexicon

Parameters:

word - The word as a String

Returns:

Whether the word is in the lexicon

tagSet

public java.util.Set<java.lang.String> tagSet(java.util.function.Function<java.lang.String,java.lang.String> basicCategoryFunction)

Return the Set of tags used by this tagger (available after training the tagger).

Specified by:

tagSet in interface Lexicon

Returns:

The Set of tags used by this tagger

ruleIteratorByWord

public java.util.Iterator<IntTaggedWord> ruleIteratorByWord(int word,
                                                            int loc,
                                                            java.lang.String featureSpec)

Description copied from interface: Lexicon

Get an iterator over all rules (pairs of (word, POS)) for this word.

Specified by:

ruleIteratorByWord in interface Lexicon

Parameters:

word - The word, represented as an integer in Index

loc - The position of the word in the sentence (counting from 0). Implementation note: The BaseLexicon class doesn't actually make use of this position information.

featureSpec - Additional word features like morphosyntactic information.

Returns:

An Iterator over a List ofIntTaggedWords, which pair the word with possible taggings as integer pairs. (Each can be thought of as a tag -> word rule.)

ruleIteratorByWord

public java.util.Iterator<IntTaggedWord> ruleIteratorByWord(java.lang.String word,
                                                            int loc,
                                                            java.lang.String featureSpec)

Description copied from interface: Lexicon

Same thing, but with a string that needs to be translated by the lexicon's word index

Specified by:

ruleIteratorByWord in interface Lexicon

numRules

public int numRules()

Returns the number of rules (tag rewrites as word) in the Lexicon. This method isn't yet implemented in this class. It currently just returns 0, which may or may not be helpful.

Specified by:

numRules in interface Lexicon

Returns:

The number of rules (tag rewrites as word) in the Lexicon.

initializeTraining

public void initializeTraining(double numTrees)

Description copied from interface: Lexicon

Start training this lexicon on the expected number of trees. (Some UnknownWordModels use the number of trees to know when to start counting statistics.)

Specified by:

initializeTraining in interface Lexicon

train

public final void train(java.util.Collection<Tree> trees)

Add the given collection of trees to the statistics counted. Can be called multiple times with different trees.

Specified by:

train in interface Lexicon

Parameters:

trees - Trees to train on

train

public void train(java.util.Collection<Tree> trees,
                  double weight)

Add the given collection of trees to the statistics counted. Can be called multiple times with different trees.

Specified by:

train in interface Lexicon

train

public void train(Tree tree,
                  double weight)

Add the given tree to the statistics counted. Can be called multiple times with different trees.

Specified by:

train in interface Lexicon

train

public void train(java.util.List<TaggedWord> sentence,
                  double weight)

Add the given sentence to the statistics counted. Can be called multiple times with different sentences.

Specified by:

train in interface Lexicon

trainUnannotated

public void trainUnannotated(java.util.List<TaggedWord> sentence,
                             double weight)

Description copied from interface: Lexicon

Sometimes we might have a sentence of tagged words which we would like to add to the lexicon, but they weren't part of a binarized, markovized, or otherwise annotated tree.

Specified by:

trainUnannotated in interface Lexicon

incrementTreesRead

public void incrementTreesRead(double weight)

Description copied from interface: Lexicon

If training on a per-word basis instead of on a per-tree basis, we will want to increment the tree count as this happens.

Specified by:

incrementTreesRead in interface Lexicon

train

public void train(TaggedWord tw,
                  int loc,
                  double weight)

Description copied from interface: Lexicon

Not all subclasses support this particular method. Those that don't will barf...

Specified by:

train in interface Lexicon

finishTraining

public void finishTraining()

Description copied from interface: Lexicon

Done collecting statistics for the lexicon.

Specified by:

finishTraining in interface Lexicon

main

public static void main(java.lang.String[] args)

score

public float score(IntTaggedWord iTW,
                   int loc,
                   java.lang.String word,
                   java.lang.String featureSpec)

Description copied from interface: Lexicon

Get the score of this word with this tag (as an IntTaggedWord) at this loc. (Presumably an estimate of P(word | tag).)

Specified by:

score in interface Lexicon

Parameters:

iTW - An IntTaggedWord pairing a word and POS tag

loc - The position in the sentence. In the default implementation this is used only for unknown words to change their probability distribution when sentence initial.

word - The word itself; useful so we don't have to look it up in an index

featureSpec - TODO

Returns:

A score, usually, log P(word|tag)

writeData

public void writeData(java.io.Writer w)
               throws java.io.IOException

Description copied from interface: Lexicon

Write the lexicon in human-readable format to the Writer. (An optional operation.)

Specified by:

writeData in interface Lexicon

Parameters:

w - The writer to output to

Throws:

java.io.IOException - If any I/O problem

readData

public void readData(java.io.BufferedReader in)
              throws java.io.IOException

Description copied from interface: Lexicon

Read the lexicon from the BufferedReader in the format written by writeData. (An optional operation.)

Specified by:

readData in interface Lexicon

Parameters:

in - The BufferedReader to read from

Throws:

java.io.IOException - If any I/O problem

getUnknownWordModel

public UnknownWordModel getUnknownWordModel()

Specified by:

getUnknownWordModel in interface Lexicon

setUnknownWordModel

public void setUnknownWordModel(UnknownWordModel uwm)

Specified by:

setUnknownWordModel in interface Lexicon

train

public void train(java.util.Collection<Tree> trees,
                  java.util.Collection<Tree> rawTrees)

Specified by:

train in interface Lexicon