|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object | +--edu.stanford.nlp.process.AbstractTokenizer | +--edu.stanford.nlp.process.SimpleTokenizer
Simple tokenizer implementation that wraps a StringTokenizer. Word delimiter chars are space and tab only. Newlines are returned as tokens. The tokens returned are edu.stanford.nlp.trees.Word objects.
Field Summary | |
protected static String |
delims
Word delimiter characters used to tokenize text: " \t\n" |
Constructor Summary | |
SimpleTokenizer()
Constructs a new SimpleTokenizer. |
|
SimpleTokenizer(Reader r)
Constructs a new SimpleTokenizer with the Reader r as its source. |
Method Summary | |
boolean |
hasNext()
Returns true if this Tokenizer has more elements. |
Object |
next()
Returns the next Word token, or null if there is none. |
void |
setSource(Reader r)
Sets the source of this Tokenizer to be the Reader r. |
Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer |
pushBack, remove, tokenize |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
protected static final String delims
Constructor Detail |
public SimpleTokenizer()
public SimpleTokenizer(Reader r)
Method Detail |
public boolean hasNext()
hasNext
in interface Tokenizer
hasNext
in class AbstractTokenizer
public Object next()
next
in interface Tokenizer
next
in class AbstractTokenizer
public void setSource(Reader r)
setSource
in interface Tokenizer
setSource
in class AbstractTokenizer
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |