T
- public static class SpanishTokenizer.SpanishTokenizerFactory<T extends HasWord> extends Object implements TokenizerFactory<T>, Serializable
Modifier and Type | Field and Description |
---|---|
protected LexedTokenFactory<T> |
factory |
protected Properties |
lexerProperties |
protected boolean |
splitCompoundOption |
protected boolean |
splitContractionOption |
protected boolean |
splitVerbOption |
Modifier and Type | Method and Description |
---|---|
Iterator<T> |
getIterator(Reader r)
Return an iterator over the contents read from r.
|
Tokenizer<T> |
getTokenizer(Reader r) |
Tokenizer<T> |
getTokenizer(Reader r,
String extraOptions) |
static TokenizerFactory<CoreLabel> |
newCoreLabelTokenizerFactory() |
static <T extends HasWord> |
newSpanishTokenizerFactory(LexedTokenFactory<T> factory,
String options)
Contructs a new SpanishTokenizer that returns T objects and uses the options passed in.
|
void |
setOptions(String options)
Set underlying tokenizer options.
|
protected final LexedTokenFactory<T extends HasWord> factory
protected Properties lexerProperties
protected boolean splitCompoundOption
protected boolean splitVerbOption
protected boolean splitContractionOption
public static TokenizerFactory<CoreLabel> newCoreLabelTokenizerFactory()
public static <T extends HasWord> SpanishTokenizer.SpanishTokenizerFactory<T> newSpanishTokenizerFactory(LexedTokenFactory<T> factory, String options)
options
- a String of options, separated by commaspublic Iterator<T> getIterator(Reader r)
IteratorFromReaderFactory
getIterator
in interface IteratorFromReaderFactory<T extends HasWord>
r
- Where to read objects frompublic Tokenizer<T> getTokenizer(Reader r)
getTokenizer
in interface TokenizerFactory<T extends HasWord>
public void setOptions(String options)
setOptions
in interface TokenizerFactory<T extends HasWord>
options
- A comma-separated list of optionspublic Tokenizer<T> getTokenizer(Reader r, String extraOptions)
getTokenizer
in interface TokenizerFactory<T extends HasWord>