This package contains implementations of three parsers for natural language text. There is an accurate unlexicalized probabilistic context-free grammar (PCFG) parser, a lexical dependency parser, and a factored, lexicalized probabilistic context free grammar parser, which does joint inference over the first two parsers. For many purposes, we recommend just using the unlexicalized PCFG. With a well-engineered grammar (as supplied for English), it is fast, accurate, requires much less memory, and in many real-world uses, lexical preferences are unavailable or inaccurate across domains or genres and it will perform just as well as a lexicalized parser. However, the factored parser will sometimes provide greater accuracy through knowledge of lexical dependencies. Using the dependency parser by itself is not very useful (its accuracy is much lower).

The factored parser and the unlexicalized PCFG parser are described in:

• Dan Klein and Christopher D. Manning. 2002. Fast Exact Inference with a Factored Model for Natural Language Parsing. Advances in Neural Information Processing Systems 15 (NIPS 2002). [pdf]
• Dan Klein and Christopher D. Manning. 2003. Accurate Unlexicalized Parsing. Proceedings of the Association for Computational Linguistics, 2003. [pdf]

Much of the internal guts of the parser are in one file, FactoredParser.java, and are not exposed as public.

The class LexicalizedParser provides an interface for either training a parser from a treebank, or parsing text using a saved parser. It can be called programmatically, or the commandline main() method supports many options.

The parser has been ported to multiple languages. German and Chinese grammars are included. The first publication below documents the Chinese parser. The German parser was developed for and used in the second paper (but the paper contains very little detail on it).

• Roger Levy and Christopher D. Manning. 2003. Is it harder to parse Chinese, or the Chinese Treebank? ACL 2003, pp. 439-446.
• Roger Levy and Christopher D. Manning. 2004. Deep dependencies from context-free statistical parsers: correcting the surface dependency approximation. ACL 2004, pp. 328-335.

End user usage

Requirements

You need Java 1.5+ installed on your system, and java in your PATH where commands are looked for.

You need a machine with a fair amount of memory. Required memory depends on the choice of parser, the size of the grammar, and other factors like presence of numerous unknown words To run the PCFG parser on sentences of up to 40 words you need 100 Mb of memory. To be able to handle longer sentences, you need more (to parse sentences up to 100 words, you need 400 Mb). For running the Factored Parser, 600 Mb is needed for dealing with sentences up to 40 words (which are quite typical in newsire!). Training a new lexicalized parser requires about 1500m of memory; much less is needed for training a PCFG.

For just parsing text, you need a saved parser model (grammars, lexicon, etc.), which can be represented either as a text file or as a binary (Java serialized object) representation. A number are provided (some compressed) (in /u/nlp/data/lexparser for local users, or in the root directory of the distributed version). For instance, there is englishFactored.ser.gz and englishPCFG.ser.gz for English, and chineseFactored.ser.gz and chinesePCFG.ser.gz for Chinese.

You need the parser code accessible. This can be done by having the supplied stanford-parser.jar in your CLASSPATH. Then if you have some sentences in testsent.txt (as plain text), the following commands should work.

Command-line parsing usage

Parsing a local text file:

java -mx100m edu.stanford.nlp.parser.lexparser.LexicalizedParser englishPCFG.ser.gz testsent.txt

Parsing a document over the web:

java -mx100m edu.stanford.nlp.parser.lexparser.LexicalizedParser -maxLength 40 englishPCFG.ser.gz http://nlp.stanford.edu/~danklein/project-parsing.shtml

Parsing a Chinese sentence (in the default input encoding of GB18030 -- and you'll need the right fonts to see the output correctly):

java -mx100m edu.stanford.nlp.parser.lexparser.LexicalizedParser -tLPP edu.stanford.nlp.parser.lexparser.ChineseTreebankParserParams chinesePCFG.ser.gz chinese-onesent

java -mx100m edu.stanford.nlp.parser.lexparser.LexicalizedParser -tLPP edu.stanford.nlp.parser.lexparser.ChineseTreebankParserParams -encoding UTF-8 chinesePCFG.ser.gz chinese-onesent-utf

The parser will send parse trees to stdout and other information on what it is doing to stderr, so one commonly wants to direct just stdout to an output file, in the standard way.

Command-line options

The program has many options. The most useful end-user option is -maxLength n which determines the maximum length sentence that the parser will parser. Longer sentences are skipped, with a message printed to stderr.

Input formatting and tokenization options

The parser supports many different input formats: tokenized/not, sentences/not, and tagged/not.

The input may be tokenized or not, and users may supply their own tokenizers. The input is by default assumed to not be tokenized; if the input is tokenized, supply the option -tokenized. If the input is not tokenized, you may supply the name of a tokenizer class with -tokenizer tokenizerClassName; otherwise the default tokenizer (edu.stanford.nlp.processor.PTBTokenizer) is used. This tokenizer should perform well over typical plain newswire-style text.

The input may have already been split into sentences or not. The input is by default assumed to be not split; if sentences are split, supply the option -sentences delimitingToken, where the delimiting token may be any string. As a special case, if the delimiting token is newline the parser will assume that each line of the file is a sentence.

The input may be in XML. The main method does not incorporate an XML parser, but one can fake certain simple cases with the -parseInside regex which will only parse the tokens inside elements matched by the regular expression regex. These elements are assumed to be pure CDATA. If you use -parseInside s, then the parser will accept input in which sentences are marked XML-style with <s> ... </s> (the same format as the input to Eugene Charniak's parser).

Finally, the input may be tagged or not. If it is tagged, the program assumes that words and tags are separated by a non-whitespace separating character such as '/' or '_'. You give the option -tagSeparator tagSeparator to specify tagged text with a tag separator.

You can see examples of many of these options in the test directory.

As an example, you can parse the example file with partial POS-tagging with this command: java edu.stanford.nlp.parser.lexparser.LexicalizedParser -maxLength 20 -sentences newline -tokenized -tagSeparator / englishPCFG.ser.gz pos-sentences.txt There are some restrictions on the interpretation of POS-tagged input:

• The tagset must match the parser POS set. If you are using our supplied parser data files, that means you must be using Penn Treebank POS tags.
• An indicated tagging will determine which of the taggings allowed by the lexicon will be used, but the parser will not accept tags not allowed by its lexicon. This is usually not problematic, since rare or unknown words are allowed to have many POS tags, but would be if you were trying to persuade it that are should be tagged as a noun in the sentence "100 are make up one hectare." since it will only allow are to have a verbal tagging.

For the examples in pos-sentences.txt:

1. This sentence is parsed correctly with no tags given.
2. So it is also parsed correctly telling the parser butter is a verb.
3. You get a different worse parse telling it butter is a noun.
4. You get the same parse as 1. with all tags correctly supplied.
5. It won't accept can as a VB, but does accept butter as a noun, so you get the same parse as 3.
6. People can butter can be an NP.
7. Most words can be NN, but not common function words like their, with, a.

For Chinese, the package includes two simple word segmenters. One is a lexicon-based maximum match segmenter, and the other uses the parser to do Hidden Markov Model-based word segmentation. These segmentation methods are okay, but if you would like a high quality segmentation of Chinese text, you will have to segment the Chinese by yourself as a preprocessing step. The supplied grammars assume that Chinese input has already been word-segmented according to Penn Chinese Treebank conventions. Choosing Chinese with -tLPP edu.stanford.nlp.parser.lexparser.ChineseTreebankParserParams makes space-separated words the default tokenization. To do word segmentation within the parser, give one of the options -segmentMarkov or -segmentMaxMatch.

Output formatting options

You can set how sentences are printed out by using the -outputFormat format option. The native and default format is as trees are formatted in the Penn Treebank, but there are a number of other useful options:

• penn The default.
• oneline Printed out on one line.
• wordsAndTags Use the parser as a POS tagger.
• latexTree Help write your LaTeX papers (for use with Avery Andrews' trees.sty package.
• typedDependenciesCollapsed Write sentences in a typed dependency format that represents sentences via grammatical relations between words. Suitable for representing text as a semantic network.

You can get each sentence printed in multiple formats by giving a comma-separated list of formats.

Programatic usage

LexicalizedParser can be easily called within a larger application. It implements a couple of useful interfaces that provide for simple use: edu.stanford.nlp.parser.ViterbiParser and edu.stanford.nlp.process.Function. The following simple class shows typical usage:

import java.util.*;
import edu.stanford.nlp.trees.*;
import edu.stanford.nlp.parser.lexparser.LexicalizedParser;

class Demo {
  public static void main(String[] args) {
    LexicalizedParser lp = new LexicalizedParser("englishPCFG.ser.gz");
    String[] sent = { "This", "is", "an", "easy", "sentence", "." };
    Tree parse = (Tree) lp.apply(Arrays.asList(sent));
    parse.pennPrint();
    System.out.println();
    TreePrint tp = new TreePrint("penn,typedDependenciesCollapsed");
    tp.print(parse);
  }
}

In a usage such as this, the parser expects sentences already tokenized according to Penn Treebank conventions. For arbitrary text, prior processing must be done to achieve such tokenization (the main method of LexicalizedParser provides an example of doing this).

Implementation notes

The current version uses class objects as temporary objects to avoid short-lived object creation and as global numberer spaces. Because of this, the parser doesn't support concurrent usage in multiple threads.

Writing and reading trained parsers to and from files

A trained parser consists of grammars, a lexicon, and option values. Once a parser has been trained, it may be written to file in one of two formats: binary serialized Java objects or human readable text data. A parser can also be quickly reconstructed (either programmatically or at the command line) from files containing a parser in either of these formats.

The binary serialized Java objects format is created using standard tools provided by the java.io package, and is not text, and not human-readable. To train and then save a parser as a binary serialized objects file, use a command line invocation of the form:

java -mx1500m edu.stanford.nlp.parser.lexparser.LexicalizedParser -train trainFilePath [fileRange] -saveToSerializedFile outputFilePath

The text data format is human readable and modifiable, and consists of four sections, appearing in the following order:

• Options - consists of variable-value pairs, one per line, which must remain constant across training and parsing.
• Lexicon - consists of lexical entries, one per line, each of which is preceded by the keyword SEEN or UNSEEN, and followed by a raw count.
• Unary Grammar - consists of unary rewrite rules, one per line, each of which is of the form A -> B, followed by the normalized log probability.
• Binary Grammar - consists of binary rewrite rules, one per line, each of which is of the form A -> B C, followed by the normalized log probability.
• Dependency Grammar

Each section is headed by a line consisting of multiple asterisks (*) and the name of the section. Note that the file format does not support rules of arbitrary arity, only binary and unary rules. To train and then save a parser as a text data file, use a command line invocation of the form:

java -mx1500m edu.stanford.nlp.parser.lexparser.LexicalizedParser -train trainFilePath start stop -saveToTextFile outputFilePath

To parse a file with a saved parser, either in text data or serialized data format, use a command line invocation of the following form:

java -mx500m edu.stanford.nlp.parser.lexparser.LexicalizedParser parserFilePath test.txt

A Note on Text Grammars

If you want to use the text grammars in another parser and duplicate our performance, you will need to know how we handle the POS tagging of rare and unknown words:

• Unknown Words: rather than scoring all words unseen during training with a single distribution over tags, we score unknown words based on their word shape signatures, defined as follows. Beginning with the original string, all lowercase alphabetic characters are replaced with x, uppercase with X, digits with d, and other characters are unchanged. Then, consecutive duplicates are eliminated. For example, Formula-1 would become Xx-1. The probability of tags given signatures is estimated on words occuring in only the second half of the training data, then inverted. However, in the current release of the parser, this is all done programatically, and so the text lexicon contains only a single UNK token. To duplicate our behavior, one would be best off building one's own lexicon with the above behavior.
• Rare Words: all words with frequency less than a cut-off (of 100) are allowed to take tags with which they were not seen during training. In this case, they are eligible for (i) all tags that either they were seen with, or (ii) any tag an unknown word can receive (lexicon entry for UNK). The probability of a tag given a rare word is an interpolation of the word's own tag distribution and the unknown distribution for that word's signature. Because of the tag-splitting used in our parser, this ability to take out-of-lexicon tags is fairly important, and not represented in our text lexicon.

For additional information

For more information, you should next look at the Javadocs for the LexicalizedParser class.