edu.stanford.nlp.process
Class DummyTokenizer

java.lang.Object
  |
  +--edu.stanford.nlp.process.AbstractTokenizer
        |
        +--edu.stanford.nlp.process.DummyTokenizer
All Implemented Interfaces:
Iterator, Tokenizer

public class DummyTokenizer
extends AbstractTokenizer

Tokenizer implementation that conforms to the Penn Treebank tokenization conventions. This tokenizer is a Java implementation of Professor Chris Manning's Flex tokenizer, pgtt-treebank.l. It reads raw text and outputs a List of tokens in the Penn treebank format. It can optionally return carriage returns as tokens.

Author:
Teg Grenager (grenager@stanford.edu)

Constructor Summary
DummyTokenizer()
           
DummyTokenizer(Reader r)
          Constructs a new DummyTokenizer that treats carriage returns as normal whitespace.
 
Method Summary
 boolean hasNext()
          Returns true if this Tokenizer has more elements.
static void main(String[] args)
          Reads a file from the argument and prints its tokens one per line.
 Object next()
          Returns the next token from this Tokenizer.
 void setSource(Reader r)
          Sets the source for this Tokenizer.
 
Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer
pushBack, remove, tokenize
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

DummyTokenizer

public DummyTokenizer()

DummyTokenizer

public DummyTokenizer(Reader r)
Constructs a new DummyTokenizer that treats carriage returns as normal whitespace.

Method Detail

hasNext

public boolean hasNext()
Description copied from class: AbstractTokenizer
Returns true if this Tokenizer has more elements.

Specified by:
hasNext in interface Tokenizer
Specified by:
hasNext in class AbstractTokenizer

next

public Object next()
Description copied from class: AbstractTokenizer
Returns the next token from this Tokenizer.

Specified by:
next in interface Tokenizer
Specified by:
next in class AbstractTokenizer

main

public static void main(String[] args)
                 throws IOException
Reads a file from the argument and prints its tokens one per line. This is mainly as a testing aid, but it can also be quite useful standalone to turn a corpus into a one token per line file of tokens.

Usage: java edu.stanford.nlp.process.DummyTokenizer filename

Parameters:
args - Command line arguments
IOException

setSource

public void setSource(Reader r)
Description copied from class: AbstractTokenizer
Sets the source for this Tokenizer.

Specified by:
setSource in interface Tokenizer
Specified by:
setSource in class AbstractTokenizer


Stanford NLP Group