A variant of the multinomial model

An alternative formalization of the represents each document $d$ as an $M$-dimensional vector of counts $\langle \termf_{t_1,d},\ldots,\termf_{t_M,d} \rangle$ where $\termf_{t_i,d}$ is the term frequency of $t_i$ in $d$. $P(d\vert\tcjclass)$ is then computed as follows (cf. Equation 99, page 12.2.1 );

$$P(d\vert\tcjclass) = P(\langle \termf_{t_1,d},\ldots,\termf_... ...ropto \prod_{1 \leq i \leq M} P(X=t_i\vert c)^{\termf_{t_i,d}}$$ (129)

Note that we have omitted the multinomial factor. See Equation 99 (page 99 ).

Equation 129 is equivalent to the sequence model in Equation 113 as $P(X=t_i\vert c)^{\termf_{t_i,d}}=1$ for terms that do not occur in $d$ ( $\termf_{t_i,d}=0$) and a term that occurs $\termf_{t_i,d} \geq 1$ times will contribute $\termf_{t_i,d}$ factors both in Equation 113 and in Equation 129.

Exercises.

• Which of the documents in Table 13.5 have identical and different bag of words representations for (i) the Bernoulli model (ii) the multinomial model? If there are differences, describe them.

• The rationale for the positional independence assumption is that there is no useful information in the fact that a term occurs in position $\tcposindex$ of a document. Find exceptions. Consider formulaic documents with a fixed document structure.

• Table 13.3 gives Bernoulli and multinomial estimates for the word the. Explain the difference.