|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectedu.stanford.nlp.process.AbstractTokenizer<HasWord>
edu.stanford.nlp.process.WordSegmentingTokenizer
public class WordSegmentingTokenizer
A tokenizer that works by calling a WordSegmenter. This is used for Chinese and Arabic.
Field Summary |
---|
Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer |
---|
nextToken |
Constructor Summary | |
---|---|
WordSegmentingTokenizer(WordSegmenter segmenter,
java.io.Reader r)
|
|
WordSegmentingTokenizer(WordSegmenter segmenter,
Tokenizer<CoreLabel> tokenizer)
|
Method Summary | |
---|---|
static TokenizerFactory<HasWord> |
factory(WordSegmenter wordSegmenter)
|
protected HasWord |
getNext()
Internally fetches the next token. |
Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer |
---|
hasNext, next, peek, remove, tokenize |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public WordSegmentingTokenizer(WordSegmenter segmenter, java.io.Reader r)
public WordSegmentingTokenizer(WordSegmenter segmenter, Tokenizer<CoreLabel> tokenizer)
Method Detail |
---|
protected HasWord getNext()
AbstractTokenizer
getNext
in class AbstractTokenizer<HasWord>
public static TokenizerFactory<HasWord> factory(WordSegmenter wordSegmenter)
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |