edu.stanford.nlp.trees
Class PennTreebankTokenizer

java.lang.Object
  edu.stanford.nlp.process.AbstractTokenizer<java.lang.String>
      edu.stanford.nlp.process.TokenizerAdapter
          edu.stanford.nlp.trees.PennTreebankTokenizer

All Implemented Interfaces:

Tokenizer<java.lang.String>, java.util.Iterator<java.lang.String>

Direct Known Subclasses:

ArabicTreebankTokenizer

public class PennTreebankTokenizer
extends TokenizerAdapter

Builds a tokenizer for English PennTreebank (release 2) trees. This is currently internally implemented via a java.io.StreamTokenizer.

Author:

Christopher Manning, Roger Levy

Field Summary

Fields inherited from class edu.stanford.nlp.process.TokenizerAdapter

eolString, st

Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer

nextToken

Constructor Summary

PennTreebankTokenizer(java.io.Reader r)

Method Summary

Methods inherited from class edu.stanford.nlp.process.TokenizerAdapter

getNext, isEol, setEolString

Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer

hasNext, next, peek, remove, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PennTreebankTokenizer

public PennTreebankTokenizer(java.io.Reader r)