edu.stanford.nlp.trees
Class AdwaitStreamTokenizer
java.lang.Object
java.io.StreamTokenizer
edu.stanford.nlp.trees.PennTagbankStreamTokenizer
edu.stanford.nlp.trees.AdwaitStreamTokenizer
- public class AdwaitStreamTokenizer
- extends PennTagbankStreamTokenizer
Builds a tokenizer for files where whitespace separates tokens,
and eol is significant. This encoding is used in Adwait-style pos
tagged files.
- Author:
- Christopher Manning
Methods inherited from class java.io.StreamTokenizer |
commentChar, eolIsSignificant, lineno, lowerCaseMode, nextToken, ordinaryChar, ordinaryChars, parseNumbers, pushBack, quoteChar, resetSyntax, slashSlashComments, slashStarComments, toString, whitespaceChars, wordChars |
AdwaitStreamTokenizer
public AdwaitStreamTokenizer(Reader r)
- Create a tokenizer for Adwait-style sentences.
This sets up simple character meanings for all non-whitespace chars
- Parameters:
r
- The reader steam
Stanford NLP Group