public static class ArabicTokenizer.ArabicTokenizerFactory<T extends HasWord> extends java.lang.Object implements TokenizerFactory<T>, java.io.Serializable
Modifier and Type | Field and Description |
---|---|
protected LexedTokenFactory<T> |
factory |
protected java.util.Properties |
lexerProperties |
Modifier and Type | Method and Description |
---|---|
java.util.Iterator<T> |
getIterator(java.io.Reader r)
Return an iterator over the contents read from r.
|
Tokenizer<T> |
getTokenizer(java.io.Reader r)
Get a tokenizer for this reader.
|
Tokenizer<T> |
getTokenizer(java.io.Reader r,
java.lang.String extraOptions)
Get a tokenizer for this reader.
|
static TokenizerFactory<CoreLabel> |
newTokenizerFactory() |
void |
setOptions(java.lang.String options)
options: A comma-separated list of options
|
protected final LexedTokenFactory<T extends HasWord> factory
protected java.util.Properties lexerProperties
public static TokenizerFactory<CoreLabel> newTokenizerFactory()
public java.util.Iterator<T> getIterator(java.io.Reader r)
IteratorFromReaderFactory
getIterator
in interface IteratorFromReaderFactory<T extends HasWord>
r
- Where to read objects frompublic Tokenizer<T> getTokenizer(java.io.Reader r)
TokenizerFactory
getTokenizer
in interface TokenizerFactory<T extends HasWord>
r
- A Reader (which is assumed to already by buffered, if appropriate)public void setOptions(java.lang.String options)
setOptions
in interface TokenizerFactory<T extends HasWord>
options
- Options for how this tokenizer should behavepublic Tokenizer<T> getTokenizer(java.io.Reader r, java.lang.String extraOptions)
TokenizerFactory
getTokenizer
in interface TokenizerFactory<T extends HasWord>
r
- A Reader (which is assumed to already by buffered, if appropriate)extraOptions
- Options for how this tokenizer should behave