edu.stanford.nlp.process

Interface Tokenizer<T>

All Superinterfaces:

java.util.Iterator<T>

All Known Implementing Classes:

AbstractTokenizer, ArabicTokenizer, ArabicTreebankTokenizer, CHTBTokenizer, FrenchTokenizer, LexerTokenizer, NegraPennTokenizer, PennTreebankTokenizer, PTBTokenizer, RobustTokenizer, SpanishTokenizer, TokenizerAdapter, WhitespaceTokenizer, WordSegmentingTokenizer

public interface Tokenizer<T>
extends java.util.Iterator<T>

Tokenizers break up text into individual Objects. These objects may be Strings, Words, or other Objects. A Tokenizer extends the Iterator interface, but provides a lookahead operation peek(). An implementation of this interface is expected to have a constructor that takes a single argument, a Reader.

Author:

Teg Grenager (grenager@stanford.edu)

Method Summary

Modifier and Type	Method and Description
T	peek() Returns the next token, without removing it, from the Tokenizer, so that the same token will be again returned on the next call to next() or peek().
java.util.List<T>	tokenize() Returns all tokens of this Tokenizer as a List for convenience.

Methods inherited from interface java.util.Iterator

forEachRemaining, hasNext, next, remove

Method Detail

peek

T peek()

Returns the next token, without removing it, from the Tokenizer, so that the same token will be again returned on the next call to next() or peek().

Returns:

the next token in the token stream.

Throws:

java.util.NoSuchElementException - If the token stream has no more tokens.

tokenize

java.util.List<T> tokenize()

Returns all tokens of this Tokenizer as a List for convenience.

Returns:

A list of all the tokens