edu.stanford.nlp.process
Interface LexedTokenFactory<T>

All Known Implementing Classes:
CoreLabelTokenFactory, WordTokenFactory

public interface LexedTokenFactory<T>

Constructs a token (of arbitrary type) from a String and its position in the underlying text. This is used to create tokens in JFlex lexers such as PTBTokenizer.


Method Summary
 T makeToken(java.lang.String str, int begin, int length)
          Constructs a token (of arbitrary type) from a String and its position in the underlying text.
 

Method Detail

makeToken

T makeToken(java.lang.String str,
            int begin,
            int length)
Constructs a token (of arbitrary type) from a String and its position in the underlying text.

Parameters:
str - The String extracted by the lexer.
begin - The offset in the document of the first character in this string.
length - The number of characters the string takes up in the document.
Returns:
The token of type T.


Stanford NLP Group