nextToken
Constructor and Description |
---|
WordSegmentingTokenizer(WordSegmenter segmenter,
Reader r) |
WordSegmentingTokenizer(WordSegmenter segmenter,
Tokenizer<CoreLabel> tokenizer) |
Modifier and Type | Method and Description |
---|---|
static TokenizerFactory<HasWord> |
factory(WordSegmenter wordSegmenter) |
protected HasWord |
getNext()
Internally fetches the next token.
|
hasNext, next, peek, remove, tokenize
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
forEachRemaining
public WordSegmentingTokenizer(WordSegmenter segmenter, Reader r)
public WordSegmentingTokenizer(WordSegmenter segmenter, Tokenizer<CoreLabel> tokenizer)
protected HasWord getNext()
AbstractTokenizer
getNext
in class AbstractTokenizer<HasWord>
public static TokenizerFactory<HasWord> factory(WordSegmenter wordSegmenter)