edu.stanford.nlp.process

Class CodepointCoreLabelProcessor

java.lang.Object

edu.stanford.nlp.process.CoreLabelProcessor

edu.stanford.nlp.process.CodepointCoreLabelProcessor

public class CodepointCoreLabelProcessor
extends CoreLabelProcessor

Processor to add codepoint counts to tokens
In general this will be the same as the character offsets, but certain fancy characters such as 𝒚̂𝒊 will change that.

Author:

John Bauer

Constructor Summary

Constructors

Constructor and Description
CodepointCoreLabelProcessor()

Method Summary

Modifier and Type	Method and Description
java.util.List<CoreLabel>	process(java.util.List<CoreLabel> tokens) Adds codepoint offsets to the tokens (parallel to character offsets).
java.util.List<CoreLabel>	restore(java.util.List<CoreLabel> originalTokens, java.util.List<CoreLabel> processedTokens) Removes the codepoint offsets in-place and returns the deannotated tokens

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CodepointCoreLabelProcessor

public CodepointCoreLabelProcessor()

Method Detail

process

public java.util.List<CoreLabel> process(java.util.List<CoreLabel> tokens)

Adds codepoint offsets to the tokens (parallel to character offsets). Does so in-place and returns the original tokens.

Specified by:

process in class CoreLabelProcessor

restore

public java.util.List<CoreLabel> restore(java.util.List<CoreLabel> originalTokens,
                                         java.util.List<CoreLabel> processedTokens)

Removes the codepoint offsets in-place and returns the deannotated tokens

Specified by:

restore in class CoreLabelProcessor