lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikita Zhiltsov <>
Subject LMDirichletSimilarity for multiple fields
Date Mon, 10 Dec 2012 05:51:00 GMT
Hi all,

I'm implementing an approach of mixture of language models in Lucene 4.0.0.
Here is a little math to be precise:

The ranking score for query q with t terms:

p(q | \theta) = \prod_{t \in q} p(t | \theta)


p(t | \theta) = \sum_f \alpha_f p(t | \theta^f)


p(t | \theta^f) = (freq(t) + \mu_f p(t | \theta_c^f)) / (length(f) + \mu_f)

\mu_f - Dirichlet prior for field f.

I've enhanced LMDirichletSimilarity to work with per-field priors:

public class LMPerFieldDirichletSimilarity extends LMDirichletSimilarity {
    protected float score(BasicStats stats, float freq, float docLen) {
        float mu = stats.getAvgFieldLength();
        float collectionProbability = ((LMStats)
        float score = (freq + mu * collectionProbability) / (docLen + mu);
        return score;

    public void computeNorm(FieldInvertState state, Norm norm) {
        byte length = new Integer(state.getLength()).byteValue();

    protected float decodeNormValue(byte norm) {
        return new Byte(norm).floatValue();

and I can mix CustomScoreQuery, BooleanQuery and FieldsQuery to get
relevant documents and compute the ranking function (the first
probability). However, my current solution omits p(t | \theta^f) values for
the fields, which do not contain occurrences of a given term t. Those
values should be computed by LMPerFieldDirichletSimilarity.score with

Surely, the problem comes from the fact that Lucene does not retrieve such
term positions by default. This problem is not so severe in case
of LMDirichletSimilarity and one-field approach, since such documents are
simply irrelevant. But in case of multi-field documents, one cannot omit
those values, if the document contains at least one term occurrence no
matter in which field.

How would you add these values while scoring?


Nikita Zhiltsov

Visiting Graduate Student
Emory University
Intelligent Information Access Lab
E500 Emerson Hall, Atlanta, Georgia, USA
Phone: (404) 834-5364

Gradute Student, Research Fellow
Kazan Federal University
Computational Linguistics Laboratory
Russia, 420008
Kazan, Prof. Nuzhina Str., 1/37 room 117
Skype: nickita.jhiltsov
Personal page:


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message