edu.stanford.nlp.process

Interface LexedTokenFactory<T>

All Known Implementing Classes:

CoreLabelTokenFactory, WordTokenFactory

public interface LexedTokenFactory<T>

Constructs a token (of arbitrary type) from a String and its position in the underlying text. This is used to create tokens in JFlex lexers such as PTBTokenizer.

Method Summary

Modifier and Type	Method and Description
T	makeToken(java.lang.String str, int begin, int length) Constructs a token (of arbitrary type) from a String and its position in the underlying text.

Method Detail

makeToken

T makeToken(java.lang.String str,
            int begin,
            int length)

Constructs a token (of arbitrary type) from a String and its position in the underlying text. The int arguments are used just to record token character offsets in an underlying text. This method does not take a substring of str.

Parameters:

str - The String extracted by the lexer.

begin - The offset in the document of the first character in this string.

length - The number of characters the string takes up in the document.

Returns:

The token of type T.