- All Implemented Interfaces:
- Tokenizer<java.lang.String>, java.util.Iterator<java.lang.String>
- Direct Known Subclasses:
- ArabicTreebankTokenizer
public class PennTreebankTokenizer
extends TokenizerAdapter
Builds a tokenizer for English PennTreebank (release 2) trees.
This is currently internally implemented via a java.io.StreamTokenizer.
- Author:
- Christopher Manning, Roger Levy