|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object | +--edu.stanford.nlp.process.AbstractTokenizer
Abstract tokenizer. Tokenizers extending AbstractTokenizer need only
implement next()
, hasNext()
, and setSource(Reader r)
.
Constructor Summary | |
AbstractTokenizer()
|
Method Summary | |
abstract boolean |
hasNext()
Returns true if this Tokenizer has more elements. |
abstract Object |
next()
Returns the next token from this Tokenizer. |
void |
pushBack()
Pushes the last token returned back on this Tokenizer, so that it will be returned again in the next call to next() . |
void |
remove()
Removes from the underlying collection the last element returned by the iterator (optional operation). |
abstract void |
setSource(Reader r)
Sets the source for this Tokenizer. |
List |
tokenize()
Returns text as a List of tokens. |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
public AbstractTokenizer()
Method Detail |
public abstract Object next()
next
in interface Tokenizer
public abstract boolean hasNext()
true
if this Tokenizer has more elements.
hasNext
in interface Tokenizer
public void remove()
Tokenizer
remove
in interface Tokenizer
public void pushBack()
Tokenizer
next()
.
pushBack
in interface Tokenizer
public List tokenize()
tokenize
in interface Tokenizer
public abstract void setSource(Reader r)
setSource
in interface Tokenizer
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |