edu.stanford.nlp.trees
Class AdwaitStreamTokenizer

java.lang.Object
  extended byjava.io.StreamTokenizer
      extended byedu.stanford.nlp.trees.PennTagbankStreamTokenizer
          extended byedu.stanford.nlp.trees.AdwaitStreamTokenizer

public class AdwaitStreamTokenizer
extends PennTagbankStreamTokenizer

Builds a tokenizer for files where whitespace separates tokens, and eol is significant. This encoding is used in Adwait-style pos tagged files.

Author:
Christopher Manning

Field Summary
 
Fields inherited from class java.io.StreamTokenizer
nval, sval, TT_EOF, TT_EOL, TT_NUMBER, TT_WORD, ttype
 
Constructor Summary
AdwaitStreamTokenizer(Reader r)
          Create a tokenizer for Adwait-style sentences.
 
Methods inherited from class java.io.StreamTokenizer
commentChar, eolIsSignificant, lineno, lowerCaseMode, nextToken, ordinaryChar, ordinaryChars, parseNumbers, pushBack, quoteChar, resetSyntax, slashSlashComments, slashStarComments, toString, whitespaceChars, wordChars
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

AdwaitStreamTokenizer

public AdwaitStreamTokenizer(Reader r)
Create a tokenizer for Adwait-style sentences. This sets up simple character meanings for all non-whitespace chars

Parameters:
r - The reader steam


Stanford NLP Group