lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <>
Subject RE: n-gram indexing
Date Sat, 30 Jul 2005 01:11:15 GMT
: Document 1:
: "united states is .... United airlines operates in 50 states. United
: states government....."
: Document 2:
: "united states is .... United airlines operates in 50 states. United
: some other word states"
: If you consider the tf-idf weight of individual terms "united" and
: "states" they would have exact score in both the documents. But the term
: "united states" should have higher weight in Document 1. The higher
: weight can be achieved through bi-grams which would include the word
: "united states". My guess is, lucene will retrieve both documents with
: equal score.

I believe yyour intuition is correct, but did you try what i suggested
below? .. i don't believe you have to index all of hte possible n-grams,
just construct a boolean query containing the n-grams, and boost the
larger n-grams (by some factor only experimenting will tell you).

you should also try out SpanNearQuery ... I believe it scores thingsw
higher if they appear closer together -- but i'm not confident of my
memory in htat case.

Seriously, try what i suggest below ... experimentation is the best way to
answer questions like this...

: -----Original Message-----
: Sent: Monday, July 18, 2005 5:11 PM
: To:
: Subject: RE: n-gram indexing
: Your intuition is right, but i can't think of any reason why you need to
: add the n-grams at indexing time -- or why using phrase queries would be
: a bad thing in this case.  When you get a multiword query, construct the
: n-grams of the query words as multiple phrase queries and search for a
: BooleanQuery of all those phrases (with bigger boosts given to longer
: phrases)
: Using your example of "united states of america" generate a query that
: looks something like...
:      united states america
:      "united states"^10 "states america"^10
:      "united states america"^100


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message