public class WhitespaceTokenizer<T extends HasWord> extends AbstractTokenizer<T>
Modifier and Type | Class and Description |
---|---|
static class |
WhitespaceTokenizer.WhitespaceTokenizerFactory<T extends HasWord>
A factory which vends WhitespaceTokenizers.
|
nextToken
Constructor and Description |
---|
WhitespaceTokenizer(LexedTokenFactory factory,
Reader r,
boolean eolIsSignificant)
Constructs a new WhitespaceTokenizer
|
Modifier and Type | Method and Description |
---|---|
static TokenizerFactory<Word> |
factory() |
static TokenizerFactory<Word> |
factory(boolean eolIsSignificant) |
protected T |
getNext()
Internally fetches the next token.
|
static void |
main(String[] args)
Reads a file from the argument and prints its tokens one per line.
|
static WhitespaceTokenizer.WhitespaceTokenizerFactory<CoreLabel> |
newCoreLabelTokenizerFactory() |
static WhitespaceTokenizer.WhitespaceTokenizerFactory<CoreLabel> |
newCoreLabelTokenizerFactory(String options) |
static WhitespaceTokenizer<CoreLabel> |
newCoreLabelWhitespaceTokenizer(Reader r) |
static WhitespaceTokenizer<CoreLabel> |
newCoreLabelWhitespaceTokenizer(Reader r,
boolean tokenizeNLs) |
static WhitespaceTokenizer<Word> |
newWordWhitespaceTokenizer(Reader r) |
static WhitespaceTokenizer<Word> |
newWordWhitespaceTokenizer(Reader r,
boolean eolIsSignificant) |
hasNext, next, peek, remove, tokenize
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
forEachRemaining
public WhitespaceTokenizer(LexedTokenFactory factory, Reader r, boolean eolIsSignificant)
r
- The Reader that is its source.eolIsSignificant
- Whether eol tokens should be returned.public static WhitespaceTokenizer.WhitespaceTokenizerFactory<CoreLabel> newCoreLabelTokenizerFactory(String options)
public static WhitespaceTokenizer.WhitespaceTokenizerFactory<CoreLabel> newCoreLabelTokenizerFactory()
protected T getNext()
getNext
in class AbstractTokenizer<T extends HasWord>
public static WhitespaceTokenizer<CoreLabel> newCoreLabelWhitespaceTokenizer(Reader r)
public static WhitespaceTokenizer<CoreLabel> newCoreLabelWhitespaceTokenizer(Reader r, boolean tokenizeNLs)
public static WhitespaceTokenizer<Word> newWordWhitespaceTokenizer(Reader r)
public static WhitespaceTokenizer<Word> newWordWhitespaceTokenizer(Reader r, boolean eolIsSignificant)
public static TokenizerFactory<Word> factory()
public static TokenizerFactory<Word> factory(boolean eolIsSignificant)
public static void main(String[] args) throws IOException
java edu.stanford.nlp.process.WhitespaceTokenizer filename
args
- Command line argumentsIOException
- If can't open files, etc.