edu.stanford.nlp.process
Interface LexedTokenFactory<T>
- All Known Implementing Classes:
- CoreLabelTokenFactory, WordTokenFactory
public interface LexedTokenFactory<T>
Constructs a token (of arbitrary type) from a String and its position
in the underlying text. This is used to create tokens in JFlex lexers
such as PTBTokenizer.
Method Summary |
T |
makeToken(String str,
int begin,
int length)
Constructs a token (of arbitrary type) from a String and its position
in the underlying text. |
makeToken
T makeToken(String str,
int begin,
int length)
- Constructs a token (of arbitrary type) from a String and its position
in the underlying text.
- Parameters:
str
- The String extracted by the lexer.begin
- The offset in the document of the first character
in this string.length
- The number of characters the string takes up in
the document.
- Returns:
- The token of type T.
Stanford NLP Group