A package for (NLP) trees, sentences, and similar things. This package provides several key abstractions (via abstract classes) and a number of further classes for related objects. Most of these classes use a Factory pattern to instantiate objects.

A Label is something that can be the label of a Tree or a Constituent. The simplest label is a StringLabel. A Word or a TaggedWord is a Label. They can be constructed with a LabelFactory. A Label often implements various interfaces, such as HasWord.

A Constituent object defines a generic edge in a graph. It has a start and end, and usually a Label. A ConstituentFactory builds a Constituent.

A Tree object provides generic facilities for manipulating NLP trees. A TreeFactory can build a Tree. A Treebank provides an interface to a collection of parsed sentences (normally found on disk as a corpus). A TreeReader reads trees from an InputStream. A TreeReaderFactory builds a TreeReader. A TreeNormalizer canonicalizes a Tree on input from a File. A HeadFinder finds the head daughter of a Tree. The TreeProcessor interface is for general sequential processing of trees, and the TreeTransformer interface is for changing them.

A Sentence is a subclass of an ArrayList. A Sentencebank provides an interface to a large number of sentences (normally found on disk as a corpus). A SentenceReader reads sentences from an InputStream. A SentenceReaderFactory builds a SentenceReader. A SentenceNormalizer canonicalizes a Sentence on input from a File. The SentenceProcessor interface is for general sequential processing of sentences.

There are also various subclasses of StreamTokenizer. The class PairFinder should probably be removed to samples.

Design notes: This package is the result of several iterations of trying to come up with a reusable and extendable set of tree classes. It may still be nonoptimal, but some thought went into it! At any rate, there are several things that it is important to understand to use the class effectively. One is that a Label has a primary value() which is always a String, and this is the only thing that matters for fundamental Label operations, such as checking equality. While anything else (or nothing) can be stored in a Label, all other Label content is regarded as purely decorative. All Label implementations should implement a labelFactory() method that returns a LabelFactory for the appropriate kind of Label. Since this depends on the exact class, this method should always be overwritten when a Label class is extended. The existing Label classes also provide a static factory() method which returns the same thing.

Road Map: There are some plans to change things. We plan to redo Label, so that all Label classes just inherit from AbstractLabel, and do a full equality test on all their fields. The default type of Treebank should be useful. TreeReader should be PennTreeReader. And there is probably more.

Illustrations of use of the trees package

Treebank and Tree

Here is some fairly straightforward code for loading trees from a treebank and iterating over the trees contained therein. It builds a histogram of sentence lengths.

import java.util.Iterator;

import edu.stanford.nlp.trees.*;
import edu.stanford.nlp.io.NumberRangesFileFilter;
import edu.stanford.nlp.util.Timing;

/** This class just prints out sentences and their lengths.
 *  Use: java SentenceLengths /turing/corpora/Treebank2/combined/wsj/07 
 *              [fileRange]
 */
public class SentenceLengths {

    private static final int maxleng = 100;
    private static int[] lengthCounts = new int[maxleng+1];
    private static int numSents = 0;


    public static void main(String[] args) {
	Timing.startTime();
	Treebank treebank = new DiskTreebank(
				     new LabeledScoredTreeReaderFactory());
	if (args.length > 1) {
	    treebank.loadPath(args[0], new NumberRangesFileFilter(args[1],
								  true));
	} else {
	    treebank.loadPath(args[0]);
	}
	
	for (Iterator it = treebank.iterator(); it.hasNext(); ) {
	    Tree t = (Tree) it.next();
	    numSents++;
	    int len = t.yield().length();
	    if (len <= maxleng) {
		lengthCounts[len]++;
	    }
	}
	System.out.print("Files " + args[0] + " ");
	if (args.length > 1) {
	    System.out.print(args[1] + " ");
	}
	System.out.println("consists of " + numSents + " sentences");
	for (int i = 0; i <= maxleng; i++) {
	    System.out.println("  " + lengthCounts[i] + " of length " + i);
	}
	Timing.endTime("Read/count all trees");
    }

}

Treebank, custom TreeReaderFactory, Tree, and Constituent

This example illustrates building a Treebank by hand, specifying a custom TreeReaderFactory, and illustrates more of the Tree package, and the notion of a Constituent. A Constituent has a start and end point and a Label.

import java.io.*;
import java.util.*;

import edu.stanford.nlp.trees.*;
import edu.stanford.nlp.util.*;

/** This class counts how often each constituent appears
 *  Use: java ConstituentCounter /turing/corpora/Treebank2/combined/wsj/07
 */
public class ConstituentCounter {

    public static void main(String[] args) {
	Treebank treebank = new DiskTreebank(new TreeReaderFactory() {
		public TreeReader newTreeReader(Reader in) {
		    return new TreeReader(in, 
			new LabeledScoredTreeFactory(new StringLabelFactory()),
				  new BobChrisTreeNormalizer());
		}
	    });

	treebank.loadPath(args[0]);
	Counter cnt = new Counter();
	
	ConstituentFactory confac = LabeledConstituent.factory();
	for (Iterator it = treebank.iterator(); it.hasNext(); ) {
	    Tree t = (Tree) it.next();
	    Set constituents = t.constituents(confac);
	    for (Iterator it2 = constituents.iterator(); it2.hasNext(); ) {
		Constituent c = (Constituent) it2.next();
		cnt.increment(c);
	    }
	}
	SortedSet ss = new TreeSet(cnt.seenSet());
	for (Iterator it = ss.iterator(); it.hasNext(); ) {
	    Constituent c = (Constituent) it.next();
	    System.out.println(c + "  " + cnt.countOf(c));
	}
    }

}

Tree and Label

Dealing with the Tree and Label classes is a central part of using this package. This code works out the set of tags (preterminal labels) used in a Treebank. It illustrates writing ones own code to recurse through a Tree, and getting a String value for a Label.

import java.util.*;
import edu.stanford.nlp.trees.*;
import edu.stanford.nlp.util.Counter;

/** This class prints out trees from strings and counts their preterminals.
 *  Use: java TreesFromStrings '(S (NP (DT This)) (VP (VBD was) (JJ good)))'
 */
public class TreesFromStrings {

    private static void addTerminals(Tree t, Counter c) {
	if (t.isLeaf()) {
	    // do nothing
	} else if (t.isPreTerminal()) {
	    c.increment(t.label().value());
	} else {
	    // phrasal node
	    Tree[] kids = t.children();
	    for (int i = 0; i < kids.length; i++) {
		addTerminals(kids[i], c);
	    }
	}
    }

    public static void main(String[] args) {
       Treebank tb = new MemoryTreebank();
       for (int i = 0; i < args.length; i++) {
	   try {
	       Tree t = Tree.valueOf(args[i]);
	       tb.add(t);
	   } catch (Exception e) {
	       e.printStackTrace();
	   }
       }
       Counter c = new Counter();
       for (Iterator it = tb.iterator(); it.hasNext(); ) {
	   Tree t = (Tree) it.next();
	   addTerminals(t, c);
       }
       System.out.println(c);
   }

}

As well as the Treebank classes, there are corresponding Sentencebank classes (though they are not quite so extensively developed. This final example shows use of a Sentencebank. It also illustrates the Visitor pattern for examining sentences in a Sentencebank. This was actually the original visitation pattern for Treebank and Sentencebank, but these days, it's in general easier to use an Iterator. You can also get Sentences from a Treebank, by taking the yield() or taggedYield() of each Tree.

import java.io.*;

import edu.stanford.nlp.trees.*;

public class SentencePrinter {

    /** Loads SentenceBank from first argument and prints it out.  

     *  Usage: java SentencePrinter sentencebankPath
     *  @param args Array of command-line arguments
     */
    public static void main(String[] args) {
	SentenceReaderFactory srf = new SentenceReaderFactory() {
		public SentenceReader newSentenceReader(Reader in) {
		    return new SentenceReader(in, new TaggedWordFactory(),
					 new PennSentenceNormalizer(),
					 new PennTagbankStreamTokenizer(in));
		}
	    };
	Sentencebank sentencebank = new DiskSentencebank(srf);
	sentencebank.loadPath(args[0]);

	sentencebank.apply(new SentenceVisitor() {
	    public void visitSentence(final Sentence s) {
		// also print tag as well as word
		System.out.println(s.toString(false));
	    }
	});
    }

}