edu.stanford.nlp.process

Class TokenizerAdapter

java.lang.Object

edu.stanford.nlp.process.AbstractTokenizer<java.lang.String>

edu.stanford.nlp.process.TokenizerAdapter

All Implemented Interfaces:

Tokenizer<java.lang.String>, java.util.Iterator<java.lang.String>

Direct Known Subclasses:

PennTreebankTokenizer

public class TokenizerAdapter
extends AbstractTokenizer<java.lang.String>

This class adapts between a java.io.StreamTokenizer and a edu.stanford.nlp.process.Tokenizer.

Version:

2004/04/01

Author:

Christopher Manning

Field Summary

Fields

Modifier and Type	Field and Description
protected java.lang.String	eolString
protected java.io.StreamTokenizer	st

Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer

NEWLINE_TOKEN, nextToken

Constructor Summary

Constructors

Constructor and Description
TokenizerAdapter(java.io.StreamTokenizer st) Create a new TokenizerAdaptor.

Method Summary

Modifier and Type	Method and Description
java.lang.String	getNext() Internally fetches the next token.
boolean	isEol(java.lang.String str) Say whether the String is the end-of-line token for this tokenizer.
void	setEolString(java.lang.String eolString) Set the String returned when the inner tokenizer returns an end-of-line token.

Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer

hasNext, next, peek, remove, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

st

protected final java.io.StreamTokenizer st

eolString

protected java.lang.String eolString

Constructor Detail

TokenizerAdapter

public TokenizerAdapter(java.io.StreamTokenizer st)

Create a new TokenizerAdaptor. In general, it is recommended that the passed in StreamTokenizer should have had resetSyntax() done to it, so that numbers are returned as entered as tokens of type String, though this code will cope as best it can.

Parameters:

st - The internal java.io.StreamTokenizer

Method Detail

getNext

public java.lang.String getNext()

Internally fetches the next token.

Specified by:

getNext in class AbstractTokenizer<java.lang.String>

Returns:

the next token in the token stream, or null if none exists.

setEolString

public void setEolString(java.lang.String eolString)

Set the String returned when the inner tokenizer returns an end-of-line token. This will only happen if the inner tokenizer has been set to eolIsSignificant(true).

Parameters:

eolString - The String used to represent eol. It is not allowed to be null (which would confuse line ends and file end)

isEol

public boolean isEol(java.lang.String str)

Say whether the String is the end-of-line token for this tokenizer.

Parameters:

str - The String being tested

Returns:

Whether it is the end-of-line token