lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From José Ramón Pérez Agüera <jose.agu...@gmail.com>
Subject Re: Integrating Language Models into Lucene
Date Thu, 26 Feb 2009 10:27:50 GMT
you have a Lucene LM implementation only for research purposes in

http://ilps.science.uva.nl/resources/lm-lucene

is a very old implementation but maybe could be useful to you

jose

On Thu, Feb 26, 2009 at 9:25 AM, Paul Elschot <paul.elschot@xs4all.nl> wrote:
> On Thursday 26 February 2009 02:21:41 Koren Krupko wrote:
>
>>
>
>> Hello Lucene Developers!
>
>>
>
>> My name is Koren Krupko. I'm quite new to Lucene but I do have experience
>> in
>
>> research in the fields of information retrieval. After reviewing Lucene's
>
>> capabilities I understand that one of its major strengths is its
>> scalability
>
>> (as opposed to other frameworks such as Lemur). However, the retrieval and
>
>> scoring models used by Lucene are based upon the rather obsolete
>> traditional
>
>> Vector Space Model. I'm interested in adding newer, state of the art,
>
>> retrieval models based on the notion of Language Models (see
>
>> http://www.nabble.com/file/p22215790/LM-review.pdf LM-review.pdf for more
>
>> details).
>
>> During the last years, retrieval systems based on LM have outperformed
>> their
>
>> VSM based counterparts consistently in well recognized competitions such
>> as
>
>> TREC. Thus, in order to make Lucene more attractive to IR researchers, I
>
>> would like to implement the following LM scoring functions using both
>
>> Jelinek-Mercer and Dirichlet priors smoothing functions: Query Likelihood,
>
>> KL-Divergence and Cross Entropy.
>
>> Integrating Language Models into Lucene in addition to its proven
>
>> performance capabilities and ease of use, will undoubtedly advance Lucene
>
>> into becoming the leading open source IR framework.
>
>>
>
>> Assuming the usage of an Inverted Index holding posting lists, in order to
>
>> implement basic LM scoring functions, I need the following information
>
>> available during query time:
>
>> 1. For each term in the inverted index –
>
>> a. Frequency in every document.
>
>> b. Frequency in the corpus.
>
>> 2. For each document – its size.
>
>> 3. Total size of the corpus.
>
>> As I understand, 1a is implemented in Lucene but the problem is getting
>> 1b,
>
>> 2 and 3 since these details are not calculated during indexing. As I see
>> it,
>
>> one could use the Payload to store document size.
>
> The field size is encoded in the norms.
>
>> However, adding the Corpus
>
>> statistics requires fundamental changes in the index file format. From
>> first
>
>> glance, this addition isn't substantial space-wise since all we need is
>> one
>
>> more parameter per term. My eventual goal is to build a more complete and
>
>> comprehensive index once that will allow running multiple sessions of
>
>> retrieval using different scoring models later.
>
>> I did a survey of the forum but didn't find anything similar to my ideas
>
>> (the closest I got was https://issues.apache.org/jira/browse/LUCENE-965).
>> I
>
>> also understand that there are thoughts regarding changing the index
>> format
>
>> in the future ("flexible indexing" -
>
>> https://issues.apache.org/jira/browse/LUCENE-1458).
>
>>
>
>> My questions are:
>
>> 1. Has anyone tried to do something similar in the past?
>
> This is a term scorer that simply divides term frequency by field length:
>
> https://issues.apache.org/jira/browse/LUCENE-293
>
> A better field length encoding would be welcome, but it's a start.
>
>> 2. Is anyone working on something similar at the moment?
>
> Me, not any more, but that's for other reasons than the qualities of LM.
>
>> 3. Do you think LM can/should become a part of official future Lucene
>
>> versions?
>
> A contrib module with an alternative set of scorers would be a nice goal,
>
> for example starting from the one referenced above.
>
>> 4. How would you recommend implementing the index additions with minimal
>
>> changes as a temporary patch?
>
> No need for a temporary patch, just create a separate issue for each index
>
> addition, and see what happens.
>
> Regards,
>
> Paul Elschot



-- 
José Ramón Pérez Agüera

Dept. de Ingeniería del Software e Inteligencia Artificial
Despacho 411 tlf. 913947599
Facultad de Informática
Universidad Complutense de Madrid

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message