The Stanford NLP (Natural Language Processing) Group

Stanford CoreNLP FAQ

Questions

What encoding does Stanford CoreNLP use?
Can you say more about adding a custom annotator?
What is the format of the XML output for coref?
CoreNLP runs out of memory?

Questions with answers

What encoding does Stanford CoreNLP use?

By default, it uses UTF-8. You can change the encoding used when reading files by setting the encoding property or by supplying the command line flag -encoding FOO.
Can you say more about adding a custom annotator?

Here are the steps:
- extend the class edu.stanford.nlp.pipeline.Annotator
  I assume you're writing your own code to do the processing. Whatever code you write, you want to call it from a class that is a subclass of Annotator. Look at any of the existing Annotator classes, such as POSTaggerAnnotator, and try to emulate what it does.
- Have a constructor with the signature (String, Properties)
  If your new annotator is FilterAnnotator, for example, it must have a constructor FilterAnnotator(String name, Properties props) in order to work.
- Add the property customAnnotatorClass.FOO=BAR
  Using the same example, suppose your full class name is com.foo.FilterAnnotator, and you want the new annotator to have the name "filter". When creating the CoreNLP properties, you need to add the flag
  customAnnotatorClass.filter=com.foo.FilterAnnotator
  You can then add "filter" to the list of annotators in the annotators property. When you do that, the constructor FilterAnnotator(String, Properties) will be called with the name "filter" and the properties files you run CoreNLP with. This lets you define any property flag you want. For example, you could name a flag filter.verbose and then extract that flag from the properties to determine the verbosity of your new annotator.
What is the format of the XML output for coref?
Here is a sample block of coref xml output:
```
    <coreference>
      <coreference>
        <mention representative="true">
          <sentence>1</sentence>
          <start>1</start>
          <end>3</end>
          <head>2</head>
        </mention>
        <mention>
          <sentence>2</sentence>
          <start>1</start>
          <end>2</end>
          <head>1</head>
        </mention>
      </coreference>
    </coreference>
```
The entire coref section is demarked by a <coreference> section. Each individual chain is then demarked by another <coreference>. (This is perhaps an unfortunate naming, but at this point there are no plans to change it.)
Inside the <coreference> section for each chain is a block describing each of the mentions. One mention will be labeled the representative mention. There are fields for sentence, indexed from 1 the range of words, from start (inclusive) to end (not inclusive), also indexed from 1, and head, the index in the sentence of the head word of this mention.
CoreNLP runs out of memory?

Either add more memory, use fewer annotators, or give CoreNLP smaller documents. Nearly all our annotators load large model files which use lots of memory. Running the full CoreNLP pipeline requires the sum of all these memory requirements. Additionally, the coreference module operates over an entire document. As the document size increases, its processing time and space increase without bound.

Stanford CoreNLP FAQ

Questions

Questions with answers

What encoding does Stanford CoreNLP use?

Can you say more about adding a custom annotator?

What is the format of the XML output for coref?

CoreNLP runs out of memory?