edu.stanford.nlp.process

Class LexerTokenizer

java.lang.Object

edu.stanford.nlp.process.AbstractTokenizer<java.lang.String>

edu.stanford.nlp.process.LexerTokenizer

All Implemented Interfaces:

Tokenizer<java.lang.String>, java.util.Iterator<java.lang.String>

Direct Known Subclasses:

NegraPennTokenizer

public class LexerTokenizer
extends AbstractTokenizer<java.lang.String>

An implementation of Tokenizer designed to work with Lexer implementing classes. Throw in a Lexer on construction and you get a Tokenizer.

Author:

Roger Levy

Field Summary

Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer

NEWLINE_TOKEN, nextToken

Constructor Summary

Constructors

Constructor and Description
LexerTokenizer(Lexer l) Constructs a tokenizer from a Lexer.
LexerTokenizer(Lexer l, java.io.Reader r) Constructs a tokenizer from a Lexer and makes a Reader the active input stream for the tokenizer.

Method Summary

Modifier and Type	Method and Description
protected java.lang.String	getNext() Internally fetches the next token.
static void	main(java.lang.String[] args) For testing only.

Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer

hasNext, next, peek, remove, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Constructor Detail

LexerTokenizer

public LexerTokenizer(Lexer l)

Constructs a tokenizer from a Lexer.

LexerTokenizer

public LexerTokenizer(Lexer l,
                      java.io.Reader r)

Constructs a tokenizer from a Lexer and makes a Reader the active input stream for the tokenizer.

Method Detail

getNext

protected java.lang.String getNext()

Internally fetches the next token.

Specified by:

getNext in class AbstractTokenizer<java.lang.String>

Returns:

the next token in the token stream, or null if none exists.

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException

For testing only.

Throws:

java.io.IOException