edu.stanford.nlp.process

Class WhitespaceTokenizer.WhitespaceTokenizerFactory<T extends HasWord>

java.lang.Object

edu.stanford.nlp.process.WhitespaceTokenizer.WhitespaceTokenizerFactory<T>

All Implemented Interfaces:

IteratorFromReaderFactory<T>, TokenizerFactory<T>

Enclosing class:

WhitespaceTokenizer<T extends HasWord>

public static class WhitespaceTokenizer.WhitespaceTokenizerFactory<T extends HasWord>
extends Object
implements TokenizerFactory<T>

A factory which vends WhitespaceTokenizers.

Author:

Christopher Manning

Constructor Summary

Constructors

Constructor and Description
WhitespaceTokenizerFactory(LexedTokenFactory<T> factory)
WhitespaceTokenizerFactory(LexedTokenFactory<T> factory, boolean tokenizeNLs)
WhitespaceTokenizerFactory(LexedTokenFactory<T> factory, String options)

Method Summary

Modifier and Type	Method and Description
Iterator<T>	getIterator(Reader r) Return an iterator over the contents read from r.
Tokenizer<T>	getTokenizer(Reader r)
Tokenizer<T>	getTokenizer(Reader r, String extraOptions)
static TokenizerFactory<Word>	newTokenizerFactory() Constructs a new TokenizerFactory that returns Word objects and treats carriage returns as normal whitespace.
void	setOptions(String options)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WhitespaceTokenizerFactory

public WhitespaceTokenizerFactory(LexedTokenFactory<T> factory)

WhitespaceTokenizerFactory

public WhitespaceTokenizerFactory(LexedTokenFactory<T> factory,
                                  String options)

WhitespaceTokenizerFactory

public WhitespaceTokenizerFactory(LexedTokenFactory<T> factory,
                                  boolean tokenizeNLs)

Method Detail

newTokenizerFactory

public static TokenizerFactory<Word> newTokenizerFactory()

Constructs a new TokenizerFactory that returns Word objects and treats carriage returns as normal whitespace. THIS METHOD IS INVOKED BY REFLECTION BY SOME OF THE JAVANLP CODE TO LOAD A TOKENIZER FACTORY. IT SHOULD BE PRESENT IN A TokenizerFactory.

Returns:

A TokenizerFactory that returns Word objects

getIterator

public Iterator<T> getIterator(Reader r)

Description copied from interface: IteratorFromReaderFactory

Return an iterator over the contents read from r.

Specified by:

getIterator in interface IteratorFromReaderFactory<T extends HasWord>

Parameters:

r - Where to read objects from

Returns:

An Iterator over the objects

getTokenizer

public Tokenizer<T> getTokenizer(Reader r)

Specified by:

getTokenizer in interface TokenizerFactory<T extends HasWord>

getTokenizer

public Tokenizer<T> getTokenizer(Reader r,
                                 String extraOptions)

Specified by:

getTokenizer in interface TokenizerFactory<T extends HasWord>

setOptions

public void setOptions(String options)

Specified by:

setOptions in interface TokenizerFactory<T extends HasWord>