edu.stanford.nlp.process
Class LexerTokenizer

java.lang.Object
  extended byedu.stanford.nlp.process.AbstractTokenizer
      extended byedu.stanford.nlp.process.LexerTokenizer
All Implemented Interfaces:
Iterator, Tokenizer

public class LexerTokenizer
extends AbstractTokenizer
implements Tokenizer

An implementation of Tokenizer designed to work with Lexer implementing classes. Throw in a Lexer on construction and you get a Tokenizer.

Author:
Roger Levy

Field Summary
 
Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer
nextToken
 
Constructor Summary
LexerTokenizer(Lexer l)
           
LexerTokenizer(Lexer l, Reader r)
           
 
Method Summary
protected  Object getNext()
           
static void main(String[] args)
          for testing only
 
Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer
hasNext, next, peek, remove, tokenize
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface edu.stanford.nlp.process.Tokenizer
hasNext, next, peek, remove, tokenize
 

Constructor Detail

LexerTokenizer

public LexerTokenizer(Lexer l)

LexerTokenizer

public LexerTokenizer(Lexer l,
                      Reader r)
Method Detail

getNext

protected Object getNext()
Specified by:
getNext in class AbstractTokenizer

main

public static void main(String[] args)
                 throws IOException
for testing only

Throws:
IOException


Stanford NLP Group