edu.stanford.nlp.process

Class WordToSentenceProcessor<IN>

java.lang.Object

edu.stanford.nlp.process.WordToSentenceProcessor<IN>

Type Parameters:

IN - The type of the tokens in the sentences

All Implemented Interfaces:

ListProcessor<IN,List<IN>>

public class WordToSentenceProcessor<IN>
extends Object
implements ListProcessor<IN,List<IN>>

Transforms a List of words into a List of Lists of words (that is, a List of sentences), by grouping the words. The word stream is assumed to already be adequately tokenized, and this class just divides the List into sentences, perhaps discarding some separator tokens as it goes.

The main behavior is to look for sentence ending tokens like "." or "?!?", and to split after them and any following sentence closers like ")". Overlaid on this is an overall choice of state: The WordToSentenceProcessor can be a non-splitter, which always returns one sentence. Otherwise, the WordToSentenceProcessor will also split based on paragraphs using one of these three states: (1) Ignore line breaks in splitting sentences, (2) Treat each line as a separate paragraph, or (3) Treat two consecutive line breaks as marking the end of a paragraph. The details of sentence breaking within paragraphs is controlled based on the following three variables:

• sentenceBoundaryTokens are tokens that are left in a sentence, but are to be regarded as ending a sentence. A canonical example is a period. If two of these follow each other, the second will be a sentence consisting of only the sentenceBoundaryToken.
• sentenceBoundaryFollowers are tokens that are left in a sentence, and which can follow a sentenceBoundaryToken while still belonging to the previous sentence. They cannot begin a sentence (except at the beginning of a document). A canonical example is a close parenthesis ')'.
• sentenceBoundaryToDiscard are tokens which separate sentences and which should be thrown away. In web documents, a typical example would be a '<p>' tag. If two of these follow each other, they are coalesced: no empty Sentence is output. The end-of-file is not represented in this Set, but the code behaves as if it were a member.
• regionElementRegex A regular expression for element names containing a sentence region. Only tokens in such elements will be included in sentences. The start and end tags themselves are not included in the sentence.

Instances of this class are now immutable. ☺

Author:

Joseph Smarr (jsmarr@stanford.edu), Christopher Manning, Teg Grenager (grenager@stanford.edu), Sarah Spikes (sdspikes@cs.stanford.edu) (Templatization)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	WordToSentenceProcessor.NewlineIsSentenceBreak

Field Summary

Fields

Modifier and Type	Field and Description
static Set<String>	DEFAULT_BOUNDARY_FOLLOWERS
static String	DEFAULT_BOUNDARY_REGEX
static Set<String>	DEFAULT_SENTENCE_BOUNDARIES_TO_DISCARD

Constructor Summary

Constructors

Constructor and Description
WordToSentenceProcessor() Create a WordToSentenceProcessor using a sensible default list of tokens for sentence ending for English/Latin writing systems.
WordToSentenceProcessor(boolean isOneSentence) Create a WordToSentenceProcessor which never breaks the input into multiple sentences.
WordToSentenceProcessor(Set<String> boundaryToDiscard) Set the set of Strings that will mark the end of a sentence, and which will be discarded after doing so.
WordToSentenceProcessor(String boundaryTokenRegex, Set<String> boundaryFollowers, Set<String> boundariesToDiscard, Set<String> xmlBreakElementsToDiscard, String regionElementRegex, WordToSentenceProcessor.NewlineIsSentenceBreak newlineIsSentenceBreak, SequencePattern<? super IN> sentenceBoundaryMultiTokenPattern, Set<String> tokenRegexesToDiscard, boolean isOneSentence, boolean allowEmptySentences) Flexibly set parameters for converting a list of tokens into sentences.
WordToSentenceProcessor(String boundaryTokenRegex, Set<String> boundaryToDiscard, Set<String> xmlBreakElementsToDiscard, WordToSentenceProcessor.NewlineIsSentenceBreak newlineIsSentenceBreak) Flexibly set the set of acceptable sentence boundary tokens, but with a default set of allowed boundary following tokens.
WordToSentenceProcessor(String boundaryTokenRegex, Set<String> boundaryToDiscard, Set<String> xmlBreakElementsToDiscard, WordToSentenceProcessor.NewlineIsSentenceBreak newlineIsSentenceBreak, SequencePattern<? super IN> sentenceBoundaryMultiTokenPattern, Set<String> tokenRegexesToDiscard) Flexibly set the set of acceptable sentence boundary tokens, but with a default set of allowed boundary following tokens.
WordToSentenceProcessor(WordToSentenceProcessor.NewlineIsSentenceBreak newlineIsSentenceBreak) Create a WordToSentenceProcessor using a sensible default list of tokens for sentence ending for English/Latin writing systems.

Method Summary

Modifier and Type	Method and Description
List<List<IN>>	process(List<? extends IN> words) Take a List (including a Sentence) of input, and return a List that has been processed in some way.
<L,F> Document<L,F,List<IN>>	processDocument(Document<L,F,IN> in)
static WordToSentenceProcessor.NewlineIsSentenceBreak	stringToNewlineIsSentenceBreak(String name)
List<List<IN>>	wordsToSentences(List<? extends IN> words) Returns a List of Lists where each element is built from a run of Words in the input Document.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BOUNDARY_REGEX

public static final String DEFAULT_BOUNDARY_REGEX

See Also:

Constant Field Values

DEFAULT_BOUNDARY_FOLLOWERS

public static final Set<String> DEFAULT_BOUNDARY_FOLLOWERS

DEFAULT_SENTENCE_BOUNDARIES_TO_DISCARD

public static final Set<String> DEFAULT_SENTENCE_BOUNDARIES_TO_DISCARD

Constructor Detail

WordToSentenceProcessor

public WordToSentenceProcessor()

Create a WordToSentenceProcessor using a sensible default list of tokens for sentence ending for English/Latin writing systems. The default set is: {".","?","!"} and any combination of ! or ?, as in !!!?!?!?!!!?!!?!!!. A sequence of two or more consecutive line breaks is taken as a paragraph break which also splits sentences. This is the usual constructor for sentence breaking reasonable text, which uses hard-line breaking, so two blank lines indicate a paragraph break.

WordToSentenceProcessor

public WordToSentenceProcessor(WordToSentenceProcessor.NewlineIsSentenceBreak newlineIsSentenceBreak)

Create a WordToSentenceProcessor using a sensible default list of tokens for sentence ending for English/Latin writing systems. The default set is: {".","?","!"} and any combination of ! or ?, as in !!!?!?!?!!!?!!?!!!. You can specify the treatment of newlines as sentence breaks as one of ignored, every newline is a sentence break, or only two or more consecutive newlines are a sentence break.

Parameters:

newlineIsSentenceBreak - Strategy for treating newlines as paragraph breaks.

WordToSentenceProcessor

public WordToSentenceProcessor(boolean isOneSentence)

Create a WordToSentenceProcessor which never breaks the input into multiple sentences. If the argument is true, the input stream is always output as one sentence. (If it is false, this is equivalent to the no argument constructor, so why use this?)

Parameters:

isOneSentence - Marker argument: true means to treat input as one sentence

WordToSentenceProcessor

public WordToSentenceProcessor(String boundaryTokenRegex,
                               Set<String> boundaryToDiscard,
                               Set<String> xmlBreakElementsToDiscard,
                               WordToSentenceProcessor.NewlineIsSentenceBreak newlineIsSentenceBreak)

Flexibly set the set of acceptable sentence boundary tokens, but with a default set of allowed boundary following tokens. Also can set sentence boundary to discard tokens and xmlBreakElementsToDiscard and set the treatment of newlines (boundaryToDiscard) as sentence ends. This one is convenient in allowing any of the first 3 arguments to be null, and then the usual defaults are substituted for it. The allowed set of boundary followers is: {")", "]", "}", "\"", "'", "''", "’", "”", "-RRB-", "-RSB-", "-RCB-", ")", "]", "}"}. The default set of discarded separator tokens includes the newline tokens used by WhitespaceLexer and PTBLexer.

Parameters:

boundaryTokenRegex - The set of boundary tokens. If null, use default.

boundaryToDiscard - The set of regex for sentence boundary tokens that should be discarded. If null, use default.

xmlBreakElementsToDiscard - xml element names like "p", which will be recognized, treated as sentence ends, and discarded. If null, use none.

newlineIsSentenceBreak - Strategy for counting line ends (boundaryToDiscard) as sentence ends.

WordToSentenceProcessor

public WordToSentenceProcessor(String boundaryTokenRegex,
                               Set<String> boundaryToDiscard,
                               Set<String> xmlBreakElementsToDiscard,
                               WordToSentenceProcessor.NewlineIsSentenceBreak newlineIsSentenceBreak,
                               SequencePattern<? super IN> sentenceBoundaryMultiTokenPattern,
                               Set<String> tokenRegexesToDiscard)

Flexibly set the set of acceptable sentence boundary tokens, but with a default set of allowed boundary following tokens. Also can set sentence boundary to discard tokens and xmlBreakElementsToDiscard and set the treatment of newlines (boundaryToDiscard) as sentence ends. This one is convenient in allowing any of the first 3 arguments to be null, and then the usual defaults are substituted for it. The allowed set of boundary followers is: {")", "]", "}", "\"", "'", "''", "’", "”", "-RRB-", "-RSB-", "-RCB-", ")", "]", "}"}. The default set of discarded separator tokens includes the newline tokens used by WhitespaceLexer and PTBLexer.

Parameters:

boundaryTokenRegex - The set of boundary tokens. If null, use default.

boundaryToDiscard - The set of regex for sentence boundary tokens that should be discarded. If null, use default.

xmlBreakElementsToDiscard - xml element names like "p", which will be recognized, treated as sentence ends, and discarded. If null, use none.

newlineIsSentenceBreak - Strategy for counting line ends (boundaryToDiscard) as sentence ends.

WordToSentenceProcessor

public WordToSentenceProcessor(Set<String> boundaryToDiscard)

Set the set of Strings that will mark the end of a sentence, and which will be discarded after doing so. This constructor is used for, and usually only for, doing one-sentence-per-line sentence splitting. Since in such cases, you generally want to strictly preserve the set of lines in the input, it preserves empty lines as empty sentences in the output.

Parameters:

boundaryToDiscard - A Set of String that will be matched with .equals() and will mark an end of sentence and be discarded.

WordToSentenceProcessor

public WordToSentenceProcessor(String boundaryTokenRegex,
                               Set<String> boundaryFollowers,
                               Set<String> boundariesToDiscard,
                               Set<String> xmlBreakElementsToDiscard,
                               String regionElementRegex,
                               WordToSentenceProcessor.NewlineIsSentenceBreak newlineIsSentenceBreak,
                               SequencePattern<? super IN> sentenceBoundaryMultiTokenPattern,
                               Set<String> tokenRegexesToDiscard,
                               boolean isOneSentence,
                               boolean allowEmptySentences)

Flexibly set parameters for converting a list of tokens into sentences. The whole enchilada.

Parameters:

boundaryTokenRegex - Tokens that match this regex will end a sentence, but are retained at the end of the sentence.

boundaryFollowers - This is a Set of String that are matched with .equals() which are allowed to be tacked onto the end of a sentence after a sentence boundary token, for example ")".

boundariesToDiscard - This is normally used for newline tokens if they are included in the tokenization. They may end the sentence (depending on the setting of newlineIsSentenceBreak), but at any rate are deleted from sentences in the output.

xmlBreakElementsToDiscard - These are elements like "p" or "sent", which will be wrapped into regex for approximate XML matching. They will be deleted in the output, and will always trigger a sentence boundary.

Method Detail

stringToNewlineIsSentenceBreak

public static WordToSentenceProcessor.NewlineIsSentenceBreak stringToNewlineIsSentenceBreak(String name)

process

public List<List<IN>> process(List<? extends IN> words)

Description copied from interface: ListProcessor

Take a List (including a Sentence) of input, and return a List that has been processed in some way.

Specified by:

process in interface ListProcessor<IN,List<IN>>

wordsToSentences

public List<List<IN>> wordsToSentences(List<? extends IN> words)

Returns a List of Lists where each element is built from a run of Words in the input Document. Specifically, reads through each word in the input document and breaks off a sentence after finding a valid sentence boundary token or end of file. Note that for this to work, the words in the input document must have been tokenized with a tokenizer that makes sentence boundary tokens their own tokens (e.g., PTBTokenizer).

Parameters:

words - A list of already tokenized words (must implement HasWord or be a String).

Returns:

A list of sentences.

See Also:

WordToSentenceProcessor(String, Set, Set, Set, String, NewlineIsSentenceBreak, SequencePattern, Set, boolean, boolean)

processDocument

public <L,F> Document<L,F,List<IN>> processDocument(Document<L,F,IN> in)