Exception in thread "main" java.lang.NoClassDefFoundError:edu/stanford/nlp/tagger/maxent/MaxentTagger?
Exception in thread "main" java.lang.NoClassDefFoundError:edu/stanford/nlp/tagger/maxent/MaxentTagger?
This means your Java CLASSPATH isn't set correctly, so the tagger (in
stanford-tagger.jar) isn't being found. See the examples in the
README.txt file for how to set the classpath with
See, for example,
for general discussion of the Java classpath.
For English (only), you can do this using the included
However, unlike for the Stanford parser, there is at present no support
for doing this automatically using options of the command-line version
of the tagger. You'd have to do it using code you write.
Use the flag "-tokenize false".
You can insert one or more tagger models into the jar file and give options to load a model from there. Here are detailed instructions.
cp stanford-postagger.jar stanford-postagger-withModel.jar
mkdir -p edu/stanford/nlp/models/pos-tagger/english-left3words
cp models/english-left3words-distsim.tagger edu/stanford/nlp/models/pos-tagger/english-left3words
jar -uf stanford-postagger-withModel.jar edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger
java -mx300m -cp stanford-postagger-withModel.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -model edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger -textFile sample-input.txt
MaxentTagger tagger = new MaxentTagger("edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger");
Yes! (This was added in version 2.0.) We provide MaxentTaggerServer as a simple example of a socket-based server using the POS tagger. With a bit of work, we're sure you can adapt this example to work in a REST, SOAP, AJAX, or whatever system. If not, pay us a lot of money, and we'll work it out for you.
If you're doing this, you may also be interested in single jar deployment. We'll use a continuation of the answer to the previous question in our example (but the two features are independent). The commands shown are for a Unix/Linux/Mac OS X system. For Windows, you reverse the slashes, etc. You start the server on some host by specifying a model and a port for it to run on:
The same class then includes a demonstration client, which you'll want to adapt to your own needs. You can invoke it like this:
java -mx300m -cp stanford-postagger-withModel.jar edu.stanford.nlp.tagger.maxent.MaxentTaggerServer -model edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger -port 2020 &
If you're running the server and client on the same machine, then you can omit the
$ java -cp stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTaggerServer -client -host nlp.stanford.edu -port 2020
Input some text and press RETURN to POS tag it, or just RETURN to finish.
I hope this'll show the server working.
I_PRP hope_VBP this_DT 'll_MD show_VB the_DT server_NN working_VBG ._.
-hostargument. You can provide other
MaxentTaggeroptions to the server invocation of
MaxentTaggerServer, such as
-outputFormat tsv, as needed.
If you run the tagger without changing how much memory you give to Java, there is a good chance you will run out of memory. This will be evident when the program terminates with an OutOfMemoryError.
Running from the command line, you need to supply a flag like
-mx1g. The number 1g is just an example;
if you do not have that much memory available, use less so your
computer doesn't start paging. For running a tagger
should be plenty; for training a complex tagger, you may need more memory.
When running from within Eclipse, follow these instructions to increase the memory given to a program being run.
Note also that the method
tokenize all the text in a reader, and put it in memory. This is okay
for reasonable-size files. However, if you have huge files, this can
consume an unbounded amount of memory. You will
need to adopt an alternate strategy where you only tokenize part of the
text at a time (e.g., perhaps a paragraph at a time)x.
The output tagged text can be produced in several styles. The tags can be separated from the words by a character, which you can specify (this is the default, with an underscore as the separator), or you can get two tab-separated columns (good for spreadsheets or the Unix
cut command), or you can get ouptput in XML. An example of each option appears below:
$ cat > short.txt This is a short sentence. So is this. $ java -cp stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/left3words-wsj-0-18.tagger -textFile short.txt -outputFormat slashTags 2> /dev/null This_DT is_VBZ a_DT short_JJ sentence_NN ._. So_RB is_VBZ this_DT ._. $ java -cp stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/left3words-wsj-0-18.tagger -textFile short.txt -outputFormat slashTags -tagSeparator \# 2> /dev/null This#DT is#VBZ a#DT short#JJ sentence#NN .#. So#RB is#VBZ this#DT .#. $ java -cp stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/left3words-wsj-0-18.tagger -textFile short.txt -outputFormat tsv 2> /dev/null This DT is VBZ a DT short JJ sentence NN . . So RB is VBZ this DT . . $ java -cp stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/left3words-wsj-0-18.tagger -textFile short.txt -outputFormat xml 2> /dev/null <sentence id="0"> <word wid="0" pos="DT">This</word> <word wid="1" pos="VBZ">is</word> <word wid="2" pos="DT">a</word> <word wid="3" pos="JJ">short</word> <word wid="4" pos="NN">sentence</word> <word wid="5" pos=".">.</word> </sentence> <sentence id="1"> <word wid="0" pos="RB">So</word> <word wid="1" pos="VBZ">is</word> <word wid="2" pos="DT">this</word> <word wid="3" pos=".">.</word> </sentence>
No! Most people who think that the tagger is slow have made the
mistake of running it
with the model
model is fairly slow. Essentially, that model is trying to
pull out all stops to maximize tagger accuracy. Speed consequently
suffers due to choices like using 4th order bidirectional tag conditioning.
In applications, we nearly always use the
english-left3words-distsim.tagger model, and we suggest you do
it's directly comparable to
the quite well known MXPOST tagger by Adwait Ratnaparkhi (both use a
second order conditioning model and maximum entropy classifiers; both
are in Java).
Compared to MXPOST, the Stanford POS Tagger running the left3words model
is both more accurate and considerably faster. Want a number?
It all depends, but on a 2008 nothing-special Intel server, it tags about
15000 words per second. This is also about 4 times faster than Tsuruoka's
C++ tagger which has an accuracy in between our left3words and
bidirectional-distsim models. The
tagger, another recent Java POS tagger, is minutely more accurate
than our best model (97.33% accuracy) but it is over 3 times slower than
our best model (and hence over 30 times slower than the
However, if speed is your paramount concern, you might want something still faster. This can be done by using a cheaper conditioning model class (you can get another 50% speed up in the Stanford POS tagger, with still little accuracy loss), using some other classifier type (an HMM-based tagger is just going to be faster than a discriminative, feature-based model like our maxent tagger), or doing more code optimization (probably more to be done here, but the current state is not so bad).
Some people also use the Stanford Parser as just a POS tagger. It's a quite accurate POS tagger, and so this is okay if you don't care about speed. But, if you do, it's not a good idea. Use the Stanford POS tagger.
You need to start with a .props file which contains options for the tagger to use. The .props files we used to create the sample taggers are included in the models directory; you can start from whichever one seems closest to the language you want to tag. For example, to train a new English tagger, start with the left3words tagger props file. To train a tagger for a western language other than English, you can consider the props files for the German or the French taggers, which are included in the full distribution. For languages using a different character set, you can start from the Chinese or Arabic props files. Or you can use the -genprops option to MaxentTagger, and it will write a sample properties file, with documentation, for you to modify. It writes it to stdout, so you'll want to save it to some file by redirecting output (usually with >). The # at the start of the line makes things a comment, so you'll want to delete the # before properties you wish to specify.
In these props files, there are two parameters you absolutely have to change. The first is the model parameter, which specifies the file which the trained model is output to (that is, it is created during the tagger training process). The other is the trainFile parameter, which specifies the file to load the training data from (data that you must provide).
You can specify input files in a few different formats. This is part of the trainFile property. To learn more about the formats you can use and what other the options mean, look at the javadoc for MaxentTagger.
In its most basic format, the training data is sentences of tagged
text. The words should be tagged by having the word and the tag
separated by the tagSeparator parameter. For example, if the
_, one of your training lines might look like
An_DT avocet_NN is_VBZ a_DT small_JJ ,_, cute_JJ bird_NN ._.
There are other options available for training files. For example, you can use tab separated blocks, where each line represents a word/tag pair and sentences are separated by blank lines. You can also specify PTB format trees, where the tags are extracted from the bottom layer of the tree.
You may want to experiment with other feature architectures for your
tagger. This is the "arch" property. Look at the javadoc for
ExtractorFrames and ExtractorFramesRare to learn what other arch
options exist. You might want to start with a basic tagger with the
This will create a tagger with features predicting the current tag from
each of the previous, current and next words (words(-1,1)), features
from each of those words represented in terms of the unicode character
classes they contain (unicodeshapes(-1,1)), bigram and
trigram tag sequence features that predict the current tag from the
previous one or two tags (order(2)), and additional features for trying to predict
the tag of rare or unknown words from the last 1, 2, 3, and 4 characters
of the word (suffix(4)).
If you are tagging English, you should almost certainly choose the model
Included in the distribution is a file,
describes all of the available models. For English, there are models
trained on WSJ PTB, which are useful for the purposes of academic
comparisons. There are also models titled "english" which are trained
on WSJ with additional training data, which are more useful for
general purpose text. There are models for other languages, as well,
such as Chinese, Arabic, etc.
The models with "english" in the name are trained on additional text corresponding to the same data the "english" parser models are trained on, with the exception of only using WSJ 2-21.
These clusters are a feature extracted from larger, untagged text which clusters the words into similar classes.
Here are the clusters currently used for English.
Run the tagger with the flags
-sentenceDelimiter newline -tokenize false
Unfortunately, we do not have a license to redistribute owlqn. This causes it to crash if you base your training file off a .props file that used owlqn internally. We do distribute our own experimental L1-regularized optimizer, though, which you can use with the option
or you can use a different optimizer, such as the L2-regularized L-BFGS optimizer
Use the shell redirect > filename
You can discuss other topics with Stanford POS Tagger developers and users by
java-nlp-user mailing list
(via a webpage). Or you can send other questions and feedback to
Site design by Bill MacCartney