|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectedu.stanford.nlp.process.AbstractTokenizer
Abstract tokenizer. Tokenizers extending AbstractTokenizer need only
implement the getNext()
method.
Field Summary | |
protected Object |
nextToken
|
Constructor Summary | |
AbstractTokenizer()
|
Method Summary | |
protected abstract Object |
getNext()
|
boolean |
hasNext()
Returns true if this Tokenizer has more elements. |
Object |
next()
Returns the next token from this Tokenizer. |
Object |
peek()
This is an optional operation, by default supported. |
void |
remove()
This is an optional operation, by default not supported. |
List |
tokenize()
Returns text as a List of tokens. |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
protected Object nextToken
Constructor Detail |
public AbstractTokenizer()
Method Detail |
protected abstract Object getNext()
public Object next()
next
in interface Tokenizer
public boolean hasNext()
true
if this Tokenizer has more elements.
hasNext
in interface Tokenizer
public void remove()
remove
in interface Tokenizer
public Object peek()
peek
in interface Tokenizer
public List tokenize()
tokenize
in interface Tokenizer
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |