edu.stanford.nlp.process.stattok

Class StatTokSentTrainer

java.lang.Object

edu.stanford.nlp.process.stattok.StatTokSentTrainer

public class StatTokSentTrainer
extends java.lang.Object

This class is used to build models for Tokenization and S. Splitting. The class can be called to train the model on any UD-formatted (CoNLL-U)
Input files:
a ud-train file

        # newdoc = tanl
        # sent_id = isst_tanl-1
        # text = LONDRA.
        1       LONDRA  Londra  PROPN   SP      _       0       root    _       SpaceAfter=No
        2       .       .       PUNCT   FS      _       1       punct   _       _

        # sent_id = isst_tanl-2
        # text = Gas dalla statua.
 

extracted features for tokenizer:

        n characters before
        n characters after
        character case
 

Field Summary

Fields

Modifier and Type	Field and Description
static java.util.Set<java.lang.String>	ARGS_TO_DROP

Constructor Summary

Constructors

Constructor and Description
StatTokSentTrainer(java.lang.String[] propertiesArguments)

Method Summary

Modifier and Type	Method and Description
java.util.List<java.lang.String>	addFeatures(java.util.ArrayList<Pair<java.lang.String,java.lang.String>> classCharsText, int windowSize) This method adds features for building the training input for the classifier.
java.util.ArrayList<Pair<java.lang.String,java.lang.String>>	fileToTrainSet(java.lang.String trainFile, java.util.Map<java.lang.String,java.lang.String[]> multiWordRules)
static void	help()
static java.util.Map<java.lang.String,java.lang.String[]>	inferMultiWordRules(java.lang.String trainFile) Method to infer multi-word token rules directly from the training set for tokenization.
static void	main(java.lang.String[] args) Main method to train the tokenizer.
static java.util.Map<java.lang.String,java.lang.String[]>	readMultiWordRules(java.lang.String multiWordRulesFile) Method to read multi-word token rules from a file.
static void	serialize(java.lang.String serializeTo, ColumnDataClassifier cdc, int windowSize)
static void	writeMultiWordRules(java.lang.String multiWordRulesFile, java.util.Map<java.lang.String,java.lang.String[]> rules)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ARGS_TO_DROP

public static final java.util.Set<java.lang.String> ARGS_TO_DROP

Constructor Detail

StatTokSentTrainer

public StatTokSentTrainer(java.lang.String[] propertiesArguments)

Method Detail

fileToTrainSet

public java.util.ArrayList<Pair<java.lang.String,java.lang.String>> fileToTrainSet(java.lang.String trainFile,
                                                                                   java.util.Map<java.lang.String,java.lang.String[]> multiWordRules)
                                                                            throws java.io.IOException,
                                                                                   java.io.FileNotFoundException

Throws:

java.io.IOException

java.io.FileNotFoundException

addFeatures

public java.util.List<java.lang.String> addFeatures(java.util.ArrayList<Pair<java.lang.String,java.lang.String>> classCharsText,
                                                    int windowSize)

This method adds features for building the training input for the classifier. Generated features are: - Before and after charachers (given the window size) - Class for the previous character - Case of the character

readMultiWordRules

public static java.util.Map<java.lang.String,java.lang.String[]> readMultiWordRules(java.lang.String multiWordRulesFile)
                                                                             throws java.io.IOException

Method to read multi-word token rules from a file.

Throws:

java.io.IOException

writeMultiWordRules

public static void writeMultiWordRules(java.lang.String multiWordRulesFile,
                                       java.util.Map<java.lang.String,java.lang.String[]> rules)
                                throws java.io.IOException

Throws:

java.io.IOException

inferMultiWordRules

public static java.util.Map<java.lang.String,java.lang.String[]> inferMultiWordRules(java.lang.String trainFile)
                                                                              throws java.io.IOException,
                                                                                     java.io.FileNotFoundException

Method to infer multi-word token rules directly from the training set for tokenization.

Throws:

java.io.IOException

java.io.FileNotFoundException

help

public static void help()

serialize

public static void serialize(java.lang.String serializeTo,
                             ColumnDataClassifier cdc,
                             int windowSize)
                      throws java.io.IOException

Throws:

java.io.IOException

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException

Main method to train the tokenizer. The training set and optionally the multi-word rules are obtained via properties. Then, a temporary training file is created. The file contains one character per line and features for the carachter. The file is then fed to the ColumnDataClassifier for training. Properties for the ColumnDataClassifier are specified in the same properties file, and follows properties of the original CoreNLP class.

Throws:

java.io.IOException