|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectedu.stanford.nlp.process.AbstractTokenizer<java.lang.String>
edu.stanford.nlp.process.TokenizerAdapter
edu.stanford.nlp.trees.PennTreebankTokenizer
public class PennTreebankTokenizer
Builds a tokenizer for English PennTreebank (release 2) trees. This is currently internally implemented via a java.io.StreamTokenizer.
Field Summary |
---|
Fields inherited from class edu.stanford.nlp.process.TokenizerAdapter |
---|
eolString, st |
Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer |
---|
nextToken |
Constructor Summary | |
---|---|
PennTreebankTokenizer(java.io.Reader r)
|
Method Summary |
---|
Methods inherited from class edu.stanford.nlp.process.TokenizerAdapter |
---|
getNext, isEol, setEolString |
Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer |
---|
hasNext, next, peek, remove, tokenize |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public PennTreebankTokenizer(java.io.Reader r)
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |