edu.stanford.nlp.process
Class LexerTokenizer

java.lang.Object
  edu.stanford.nlp.process.AbstractTokenizer<java.lang.String>
      edu.stanford.nlp.process.LexerTokenizer

All Implemented Interfaces:

Tokenizer<java.lang.String>, java.util.Iterator<java.lang.String>

Direct Known Subclasses:

NegraPennTokenizer

public class LexerTokenizer
extends AbstractTokenizer<java.lang.String>

An implementation of Tokenizer designed to work with Lexer implementing classes. Throw in a Lexer on construction and you get a Tokenizer.

Author:

Roger Levy

Field Summary

Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer

nextToken

Constructor Summary

LexerTokenizer(Lexer l)

LexerTokenizer(Lexer l, java.io.Reader r)

Method Summary

protected java.lang.String	getNext() Internally fetches the next token.
static void	main(java.lang.String[] args) for testing only

Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer

hasNext, next, peek, remove, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LexerTokenizer

public LexerTokenizer(Lexer l)

LexerTokenizer

public LexerTokenizer(Lexer l,
                      java.io.Reader r)

Method Detail

getNext

protected java.lang.String getNext()

Internally fetches the next token.

Specified by:

getNext in class AbstractTokenizer<java.lang.String>

Returns:

the next token in the token stream, or null if none exists.

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException

for testing only

Throws:

java.io.IOException