|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectedu.stanford.nlp.process.AbstractTokenizer<java.lang.String>
edu.stanford.nlp.process.LexerTokenizer
public class LexerTokenizer
An implementation of Tokenizer
designed to work with
Lexer
implementing classes. Throw in a Lexer
on
construction and you get a Tokenizer
.
Field Summary |
---|
Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer |
---|
nextToken |
Constructor Summary | |
---|---|
LexerTokenizer(Lexer l)
|
|
LexerTokenizer(Lexer l,
java.io.Reader r)
|
Method Summary | |
---|---|
protected java.lang.String |
getNext()
Internally fetches the next token. |
static void |
main(java.lang.String[] args)
for testing only |
Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer |
---|
hasNext, next, peek, remove, tokenize |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public LexerTokenizer(Lexer l)
public LexerTokenizer(Lexer l, java.io.Reader r)
Method Detail |
---|
protected java.lang.String getNext()
getNext
in class AbstractTokenizer<java.lang.String>
public static void main(java.lang.String[] args) throws java.io.IOException
java.io.IOException
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |