edu.stanford.nlp.process
Class AbstractTokenizer

java.lang.Object
  extended byedu.stanford.nlp.process.AbstractTokenizer
All Implemented Interfaces:
Iterator, Tokenizer
Direct Known Subclasses:
DummyTokenizer, LexerTokenizer, PTBTokenizer, SimpleTokenizer

public abstract class AbstractTokenizer
extends Object
implements Tokenizer

Abstract tokenizer. Tokenizers extending AbstractTokenizer need only implement the getNext() method.

Author:
Teg Grenager (grenager@stanford.edu)

Field Summary
protected  Object nextToken
           
 
Constructor Summary
AbstractTokenizer()
           
 
Method Summary
protected abstract  Object getNext()
           
 boolean hasNext()
          Returns true if this Tokenizer has more elements.
 Object next()
          Returns the next token from this Tokenizer.
 Object peek()
          This is an optional operation, by default supported.
 void remove()
          This is an optional operation, by default not supported.
 List tokenize()
          Returns text as a List of tokens.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

nextToken

protected Object nextToken
Constructor Detail

AbstractTokenizer

public AbstractTokenizer()
Method Detail

getNext

protected abstract Object getNext()

next

public Object next()
Returns the next token from this Tokenizer.

Specified by:
next in interface Tokenizer

hasNext

public boolean hasNext()
Returns true if this Tokenizer has more elements.

Specified by:
hasNext in interface Tokenizer

remove

public void remove()
This is an optional operation, by default not supported.

Specified by:
remove in interface Tokenizer

peek

public Object peek()
This is an optional operation, by default supported.

Specified by:
peek in interface Tokenizer

tokenize

public List tokenize()
Returns text as a List of tokens.

Specified by:
tokenize in interface Tokenizer


Stanford NLP Group