edu.stanford.nlp.io
Class StreamTokenizerAdapter

java.lang.Object
  extended byedu.stanford.nlp.io.StreamTokenizerAdapter
All Implemented Interfaces:
StreamTokenizer
Direct Known Subclasses:
PennTreebankStreamTokenizer

public class StreamTokenizerAdapter
extends Object
implements StreamTokenizer

This class adapts between a java.io.StreamTokenizer and a edu.stanford.nlp.io.StreamTokenizer.

Author:
Christopher Manning

Constructor Summary
StreamTokenizerAdapter(StreamTokenizer st)
          Create a new StreamTokenizer.
 
Method Summary
 boolean isEol(String str)
          Say whether the String is the end-of-line token for this tokenizer.
 String next()
          Returns the next token from the input Reader.
 void pushBack()
          Pushback the last read token from the tokenizer for re-reading.
 void setEolString(String eolString)
          Set the String returned when the inner tokenizer returns an end-of-line token.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

StreamTokenizerAdapter

public StreamTokenizerAdapter(StreamTokenizer st)
Create a new StreamTokenizer. In general, it is recommended that the passed in StreamTokenizer should have had resetSyntax() done to it, so that numbers are returned as entered as of type String, though this code will cope as best it can.

Parameters:
st - The internal java.io.StreamTokenizer
Method Detail

next

public String next()
            throws IOException
Returns the next token from the input Reader. At end of file, it returns null.

Specified by:
next in interface StreamTokenizer
Returns:
The token
Throws:
IOException - If there is some I/O error

pushBack

public void pushBack()
Pushback the last read token from the tokenizer for re-reading. That is, the next call to next() will return the same token as the last call to next(). This allows just a single level of lookahead in token reading.

Specified by:
pushBack in interface StreamTokenizer

setEolString

public void setEolString(String eolString)
Set the String returned when the inner tokenizer returns an end-of-line token. This will only happen if the inner tokenizer has been set to eolIsSignificant(true).

Parameters:
eolString - The String used to represent eol. It is not allowed to be null (which would confuse line ends and file end)

isEol

public boolean isEol(String str)
Say whether the String is the end-of-line token for this tokenizer.

Parameters:
str - The String being tested
Returns:
Whether it is the end-of-line token


Stanford NLP Group