|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object | +--edu.stanford.nlp.process.AbstractTokenizer | +--edu.stanford.nlp.process.DummyTokenizer
Tokenizer implementation that conforms to the Penn Treebank tokenization conventions. This tokenizer is a Java implementation of Professor Chris Manning's Flex tokenizer, pgtt-treebank.l. It reads raw text and outputs a List of tokens in the Penn treebank format. It can optionally return carriage returns as tokens.
Constructor Summary | |
DummyTokenizer()
|
|
DummyTokenizer(Reader r)
Constructs a new DummyTokenizer that treats carriage returns as normal whitespace. |
Method Summary | |
boolean |
hasNext()
Returns true if this Tokenizer has more elements. |
static void |
main(String[] args)
Reads a file from the argument and prints its tokens one per line. |
Object |
next()
Returns the next token from this Tokenizer. |
void |
setSource(Reader r)
Sets the source for this Tokenizer. |
Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer |
pushBack, remove, tokenize |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
public DummyTokenizer()
public DummyTokenizer(Reader r)
Method Detail |
public boolean hasNext()
AbstractTokenizer
true
if this Tokenizer has more elements.
hasNext
in interface Tokenizer
hasNext
in class AbstractTokenizer
public Object next()
AbstractTokenizer
next
in interface Tokenizer
next
in class AbstractTokenizer
public static void main(String[] args) throws IOException
Usage: java edu.stanford.nlp.process.DummyTokenizer filename
args
- Command line arguments
IOException
public void setSource(Reader r)
AbstractTokenizer
setSource
in interface Tokenizer
setSource
in class AbstractTokenizer
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |