lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Elschot <paul.elsc...@xs4all.nl>
Subject Re: Phrase query vs span query
Date Wed, 22 Feb 2006 08:36:18 GMT
On Wednesday 22 February 2006 00:45, Rajesh Munavalli wrote:
> I am trying to adopt lucene for a special IR system. The following scenario
> is an approximation of what I am trying to do. Please bear with me if some
> things doesnt make sense. I need some suggestions on formulating queries for
> the following scenario
> 
> Each document consists of a set of fields (standard in lucene). But in my
> case, the field is somewhat different as explained below.
> 
> Field:
> ---------
> Each field consists of a set of conceptual sections. Each of these sections
> is separated by say N (say 1000) index positions but are in the same field.
> Sizes of sections vary and do not have any lower or upper bound on the
> number of terms they may contain
> .
> Ex: Lets say Field "contents" has
> <section1 of 100 terms><gap of 1000 term positions><section 2 of 1500
> terms><gap of 1000 term positions><gap of 1000 term positions><section
3 of
> 10 terms>
> 
> NOTE: At index time, I am assuming I somehow know how to form these
> sections.

One more choice you have is too index both the full document and each section
as a Lucene document.

> 
> Typical Query:
> ---------------------
> Consists of 15 to 30 query terms. In other words, these query terms
> represent a conceptual section.

Would you need synonyms of these terms, too?
 
> Aim of the Query formation:
> ----------------------------------------
> I want to rank the documents proportional to the number query terms

For this there is the coord() factor used in Lucene boolean queries.
But scoring exactly proportional to the number of query terms is difficult
to do because the lucene score is not bound by default.

> appearing in the SAME SECTION and IN ORDER. Documents containing terms with

To query the exact order, you can use PhraseQuery and SpanQuery.

> the
> 
> My Questions:
> ---------------------
> Considering the structure of the fields/documents and the number of query
> terms.
> 
> (1) Is there an effective way of formulating a query with the existing query
> types in Lucene?

I don't think so, see below.

> (2) After considering the way different queries work and their limitations,
> I think forming phrase/span queries of groups of query terms
> might approximate the rankings I am expecting. In that case which of the
> following queries will perform better (in terms of QUERY SPEED and RANKING)
>               (a) phrase query with certain slope factor
>               (b) span query

SpanQuery is slower than PhraseQuery, but it has the advantage that it can
be nested. Nesting here means the possibility to use eg. a short phrase as
a unit to be matched and scored.

Concerning this:
>Rank 2: Documents containing section containing all terms but randomly
>ordered

SpanQuery can also match unordered occurrences, I don't know about
PhraseQuery.

To formulate a single query for your requirements,
there is still the problem that PhraseQuery and SpanQuery only work when
all their "terms" are present in an indexed lucene document field.
Putting it differently, when fewer terms present, their order cannot
be taken into account, unless the query contains an (non)ordered query 
specifying a subset of the terms present in the documents.

An alternative to the current span query implementation is here:
http://issues.apache.org/jira/browse/LUCENE-413
but this will only help to get an impression of how to match in the ordered
and unordered cases.
It might be possible to generalize the various span algorithms there and
in the trunk to work with fewer "terms".

Regards,
Paul Elschot

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message