public class LexerTokenizer extends AbstractTokenizer<java.lang.String>
Tokenizer
designed to work with
Lexer
implementing classes. Throw in a Lexer
on
construction and you get a Tokenizer
.NEWLINE_TOKEN, nextToken
Constructor and Description |
---|
LexerTokenizer(Lexer l)
Constructs a tokenizer from a
Lexer . |
LexerTokenizer(Lexer l,
java.io.Reader r)
Constructs a tokenizer from a
Lexer and makes a Reader
the active input stream for the tokenizer. |
Modifier and Type | Method and Description |
---|---|
protected java.lang.String |
getNext()
Internally fetches the next token.
|
static void |
main(java.lang.String[] args)
For testing only.
|
hasNext, next, peek, remove, tokenize
protected java.lang.String getNext()
getNext
in class AbstractTokenizer<java.lang.String>
public static void main(java.lang.String[] args) throws java.io.IOException
java.io.IOException