|CoreTokenFactory<IN extends CoreMap>||
To make tokens like CoreMap or CoreLabel.
Top-level interface for transforming Documents.
Constructs a token (of arbitrary type) from a String and its position in the underlying text.
An interface for things that operate on a List.
This interface is a conjunction of Function and Serializable, which is a bad idea from the perspective of the type system, but one that seems more palatable than other bad ideas until java's type system is flexible enough to support type conjunctions.
Tokenizers break up text into individual Objects.
A TokenizerFactory is a factory that can build a Tokenizer (an extension of Iterator) from a java.io.Reader.
An interface for segmenting strings into words (in unwordsegmented languages).
An abstract tokenizer.
Takes a HasWord or String and returns an Americanized version of it.
Convert a Chinese Document into a List of sentence Strings.
Processor to add codepoint counts to tokens
In general this will be the same as the character offsets, but certain fancy characters such as 𝒚̂𝒊 will change that.
Abstract class for processing a
Maps a String to its distributional similarity class.
Produces a list of sentences from either a plain text or XML document.
This class contains various static utility methods invoked by our JFlex NL lexers.
Morphology computes the base form of English words, by removing just inflections (not derivational morphology).
|PTBEscapingProcessor<IN extends HasWord,L,F>||
Produces a new Document of Words in which special characters of the PTB have been properly escaped.
|PTBTokenizer<T extends HasWord>||
A fast, rule-based tokenizer implementation, which produces Penn Treebank style tokenization of English text.
|PTBTokenizer.PTBTokenizerFactory<T extends HasWord>||
This class provides a factory which will vend instances of PTBTokenizer which wrap a provided Reader.
This class adapts between a
Reads XML from an input file or stream and writes XML to an output file or stream, while transforming text appearing inside specified XML tags by applying a specified
This version of the SAXInterface doesn't escape the text produced by the function.
Reads sentences from a TSV, provided a list of fields to populate.
A set of utilities for parsing TSV files into CoreMaps
|WhitespaceTokenizer<T extends HasWord>||
A WhitespaceTokenizer is a tokenizer that splits on and discards only whitespace characters.
|WhitespaceTokenizer.WhitespaceTokenizerFactory<T extends HasWord>||
A factory which vends WhitespaceTokenizers.
A tokenizer that works by calling a WordSegmenter.
Provides static methods which map any String to another String indicative of its "word shape" -- e.g., whether capitalized, numeric, etc.
Constructs a Word from a String.
Transforms a List of words into a List of Lists of words (that is, a List of sentences), by grouping the words.
A list of possible fields in the sentence table
Document process(Document)method which takes a document and returns another document, which may be parsed, stoplisted, stemmed, etc.
Stanford NLP Group