edu.stanford.nlp.process

Class WordSegmentingTokenizer

java.lang.Object

edu.stanford.nlp.process.AbstractTokenizer<HasWord>

edu.stanford.nlp.process.WordSegmentingTokenizer

All Implemented Interfaces:

Tokenizer<HasWord>, java.util.Iterator<HasWord>

public class WordSegmentingTokenizer
extends AbstractTokenizer<HasWord>

A tokenizer that works by calling a WordSegmenter. This is used for Chinese and Arabic.

Author:

Galen Andrew, Spence Green

Field Summary

Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer

NEWLINE_TOKEN, nextToken

Constructor Summary

Constructors

Constructor and Description
WordSegmentingTokenizer(WordSegmenter segmenter, java.io.Reader r)
WordSegmentingTokenizer(WordSegmenter segmenter, Tokenizer<CoreLabel> tokenizer)

Method Summary

Modifier and Type	Method and Description
static TokenizerFactory<HasWord>	factory(WordSegmenter wordSegmenter)
protected HasWord	getNext() Internally fetches the next token.

Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer

hasNext, next, peek, remove, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Constructor Detail

WordSegmentingTokenizer

public WordSegmentingTokenizer(WordSegmenter segmenter,
                               java.io.Reader r)

WordSegmentingTokenizer

public WordSegmentingTokenizer(WordSegmenter segmenter,
                               Tokenizer<CoreLabel> tokenizer)

Method Detail

getNext

protected HasWord getNext()

Description copied from class: AbstractTokenizer

Internally fetches the next token.

Specified by:

getNext in class AbstractTokenizer<HasWord>

Returns:

the next token in the token stream, or null if none exists.

factory

public static TokenizerFactory<HasWord> factory(WordSegmenter wordSegmenter)