edu.stanford.nlp.io
Interface StreamTokenizer

All Known Implementing Classes:
AbstractStreamTokenizer, CHTBTokenizer, PennTreebankStreamTokenizer, StreamTokenizerAdapter

public interface StreamTokenizer

A simple interface for streaming tokenizers. This assumes that a token stream can be regarded as a sequence of things of type String which end at the end of file. It avoids all the complexities of the java.io.StreamTokenizer class.

The expected constructors are one that takes a Reader, perhaps optionally one that takes an InputStream, and does the composition for the user. There may also be other optional arguments that specify such things as whether eol is significant, or whether tokens should be lowercased.

Author:
Christopher Manning

Method Summary
 String next()
          Returns the next token from the input Reader.
 void pushBack()
          Pushback the last read token from the tokenizer for re-reading.
 

Method Detail

next

public String next()
            throws IOException
Returns the next token from the input Reader. At end of file, it returns null.

Returns:
The token
Throws:
IOException - If there is some I/O error

pushBack

public void pushBack()
Pushback the last read token from the tokenizer for re-reading. That is, the next call to next() will return the same token as the last call to next(). This allows just a single level of lookahead in token reading.



Stanford NLP Group