edu.stanford.nlp.process
Class PTBTokenizer

java.lang.Object
  extended byedu.stanford.nlp.process.AbstractTokenizer
      extended byedu.stanford.nlp.process.PTBTokenizer
All Implemented Interfaces:
Iterator, Tokenizer

public class PTBTokenizer
extends AbstractTokenizer

Tokenizer implementation that conforms to the Penn Treebank tokenization conventions. This tokenizer is a Java implementation of Professor Chris Manning's Flex tokenizer, pgtt-treebank.l. It reads raw text and outputs tokens as edu.stanford.nlp.trees.Words in the Penn treebank format. It can optionally return carriage returns as tokens.

Author:
Teg Grenager (grenager@stanford.edu)

Field Summary
 
Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer
nextToken
 
Constructor Summary
PTBTokenizer()
          Constructs a new PTBTokenizer that treats carriage returns as normal whitespace.
PTBTokenizer(boolean tokenizeCRs)
          Constructs a new PTBTokenizer that optionally returns carriage returns as their own token.
PTBTokenizer(Reader r)
          Constructs a new PTBTokenizer that treats carriage returns as normal whitespace.
PTBTokenizer(Reader r, boolean tokenizeCRs)
          Constructs a new PTBTokenizer that optionally returns carriage returns as their own token.
 
Method Summary
static TokenizerFactory factory()
           
static TokenizerFactory factory(boolean tokenizeCRs)
           
protected  Object getNext()
          Get the next valid Word from the lexer if possible.
static void main(String[] args)
          Reads a file from the argument and prints its tokens one per line.
static String ptb2Text(List ptbWords)
          Returns a presentable version of the given PTB-tokenized words.
static String ptb2Text(String ptbText)
          Returns a presentable version of the given PTB-tokenized text.
 void setSource(Reader r)
          Sets the source of this Tokenizer to be the Reader r.
 
Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer
hasNext, next, peek, remove, tokenize
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

PTBTokenizer

public PTBTokenizer()
Constructs a new PTBTokenizer that treats carriage returns as normal whitespace. No source is specified, so hasNext() will return false.


PTBTokenizer

public PTBTokenizer(boolean tokenizeCRs)
Constructs a new PTBTokenizer that optionally returns carriage returns as their own token. CRs come back as Words whose text is the value of PTBLexer.cr.


PTBTokenizer

public PTBTokenizer(Reader r)
Constructs a new PTBTokenizer that treats carriage returns as normal whitespace.


PTBTokenizer

public PTBTokenizer(Reader r,
                    boolean tokenizeCRs)
Constructs a new PTBTokenizer that optionally returns carriage returns as their own token. CRs come back as Words whose text is the value of PTBLexer.cr.

Method Detail

getNext

protected Object getNext()
Get the next valid Word from the lexer if possible.

Specified by:
getNext in class AbstractTokenizer

main

public static void main(String[] args)
                 throws IOException
Reads a file from the argument and prints its tokens one per line. This is mainly as a testing aid, but it can also be quite useful standalone to turn a corpus into a one token per line file of tokens.

Usage: java edu.stanford.nlp.process.PTBTokenizer filename

Parameters:
args - Command line arguments
Throws:
IOException

setSource

public void setSource(Reader r)
Sets the source of this Tokenizer to be the Reader r.


ptb2Text

public static String ptb2Text(String ptbText)
Returns a presentable version of the given PTB-tokenized text. PTB tokenization splits up punctuation and does various other things that makes simply joining the tokens with spaces look bad. So join the tokens with space and run it through this method to produce nice looking text. It's not perfect, but it works pretty well.


ptb2Text

public static String ptb2Text(List ptbWords)
Returns a presentable version of the given PTB-tokenized words. Pass in a List of Words or Strings, or a Document and this method will join the words with spaces and call ptb2Text(String) on the output. This method will check if the elements in the list are subtypes of Word, and if so, it will take the word() values to prevent additional text from creeping in (e.g., POS tags). Otherwise the toString value will be used.


factory

public static TokenizerFactory factory()

factory

public static TokenizerFactory factory(boolean tokenizeCRs)


Stanford NLP Group