edu.stanford.nlp.trees.international.arabic
Class ArabicTreebankTokenizer

java.lang.Object
  extended by edu.stanford.nlp.process.AbstractTokenizer
      extended by edu.stanford.nlp.process.TokenizerAdapter
          extended by edu.stanford.nlp.trees.PennTreebankTokenizer
              extended by edu.stanford.nlp.trees.international.arabic.ArabicTreebankTokenizer
All Implemented Interfaces:
Tokenizer, Iterator

public class ArabicTreebankTokenizer
extends PennTreebankTokenizer

Builds a tokenizer for English PennTreebank (release 2) trees. This is currently internally implemented via a java.io.StreamTokenizer.

Author:
Christopher Manning

Field Summary
 
Fields inherited from class edu.stanford.nlp.process.TokenizerAdapter
eolString, st
 
Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer
nextToken
 
Constructor Summary
ArabicTreebankTokenizer(Reader r)
           
 
Method Summary
 Object getNext()
          Internally fetches the next token.
static void main(String[] args)
           
 
Methods inherited from class edu.stanford.nlp.process.TokenizerAdapter
isEol, setEolString
 
Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer
hasNext, next, peek, remove, tokenize
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

ArabicTreebankTokenizer

public ArabicTreebankTokenizer(Reader r)
Method Detail

getNext

public Object getNext()
Internally fetches the next token.

Overrides:
getNext in class TokenizerAdapter
Returns:
the next token in the token stream, or null if none exists.

main

public static void main(String[] args)
                 throws IOException
Throws:
IOException


Stanford NLP Group