edu.stanford.nlp.process
Class AbstractTokenizer

java.lang.Object
  |
  +--edu.stanford.nlp.process.AbstractTokenizer
All Implemented Interfaces:
Iterator, Tokenizer
Direct Known Subclasses:
DummyTokenizer, PTBTokenizer, SimpleTokenizer

public abstract class AbstractTokenizer
extends Object
implements Tokenizer

Abstract tokenizer. Tokenizers extending AbstractTokenizer need only implement next(), hasNext(), and setSource(Reader r).

Author:
Teg Grenager (grenager@stanford.edu)

Constructor Summary
AbstractTokenizer()
           
 
Method Summary
abstract  boolean hasNext()
          Returns true if this Tokenizer has more elements.
abstract  Object next()
          Returns the next token from this Tokenizer.
 void pushBack()
          Pushes the last token returned back on this Tokenizer, so that it will be returned again in the next call to next().
 void remove()
          Removes from the underlying collection the last element returned by the iterator (optional operation).
abstract  void setSource(Reader r)
          Sets the source for this Tokenizer.
 List tokenize()
          Returns text as a List of tokens.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

AbstractTokenizer

public AbstractTokenizer()
Method Detail

next

public abstract Object next()
Returns the next token from this Tokenizer.

Specified by:
next in interface Tokenizer

hasNext

public abstract boolean hasNext()
Returns true if this Tokenizer has more elements.

Specified by:
hasNext in interface Tokenizer

remove

public void remove()
Description copied from interface: Tokenizer
Removes from the underlying collection the last element returned by the iterator (optional operation). This method can be called only once per call to next.

Specified by:
remove in interface Tokenizer

pushBack

public void pushBack()
Description copied from interface: Tokenizer
Pushes the last token returned back on this Tokenizer, so that it will be returned again in the next call to next().

Specified by:
pushBack in interface Tokenizer

tokenize

public List tokenize()
Returns text as a List of tokens.

Specified by:
tokenize in interface Tokenizer

setSource

public abstract void setSource(Reader r)
Sets the source for this Tokenizer.

Specified by:
setSource in interface Tokenizer


Stanford NLP Group