Probabilistic Part of Speech Tagging


Part-of-speech tagging is assigning the correct part of speech (noun, verb, etc.) to words. We have worked on building probabilistic conditional log-linear models for tagging. Our best-performing part of speech tagger for English uses both preceding and following tag context, and many lexical features. Its accuracy is state of the art for tagging Penn Treebank. A model for Chinese has also been developed on the basis of the Toutanova et al. (2003) work.



A java implementation of The Stanford Tagger is available online.


  • Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003.Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network. In Proceedings of HLT-NAACL 2003 pages 252-259. [pdf]
  • Kristina Toutanova and Christopher D. Manning. Enriching the Knowledge Sources Used in a Maximum Entropy Part-of-Speech Tagger. Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP/VLC-2000), Hong Kong.[pdf]