lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antoine Baudoux ...@taktik.be>
Subject Re: Several questions about scoring/sorting + random sorting in an image/related application
Date Fri, 15 Jun 2007 20:37:59 GMT

>
> On 15 Jun 2007, at 19:07, Walt Stoneburner wrote:
> Antoine Baudoux writes:
> I want to be able to give a score to each collection.
>
> Keep in mind, Lucene is computing a score based on quite a number of
> things from how often a term is used in a document, how often it
> appears in the collection of documents, how long the query is, etc.
>
> If your concept of a document's score changes, then I'd be inclined to
> think you're possibly using Lucene in a manner it wasn't designed for.
> That said, I have two thoughts.
>
> THOUGHT ONE
> Use Lucene to locate "records" for you --- what you really are
> interested in getting back _from Lucene_ is the primary key.  Then,
> use this key to do a lookup in your database of the score of the day
> and sort accordingly.  The idea is that Lucene finds, your table
> scores, and because of that you won't need to re-index when something
> changes.


I think Thats the same principle that drives ValueSourceQuery. The  
advantadge of ValueSourceQuery is that the sorting remains in lucene,  
where it can benefits of field indexes and Field Caches.

  I dont like the idea of having to retrieve, say, 300.000 results  
from lucene, then sort them using lookups in DB, then display only a  
part of the results on screen. But I may be wrong thinking it would  
be slower than a ValueSourceQuery.
>
>
> THOUGHT TWO
> Use boosting.  COLLECTION_ONE^5  COLLECTION_THREE^10  etc.  That way
> /if/ the Lucene document appears in the collection, it's score is
> weighted according to your preferences.  You're free to change the
> boosts on a query-by-query basis without having to re-index.

> I can use a Very big ... query ... I am afraid that it will be slow.

> Try it.  I think you'll find Lucene is _fast_.  We do some pretty HUGE
> and complicated queries and Lucene just screams.
>

Indeed, I tried a Boolean query with a custom boost for each of the  
300 collections, the speed is correct with a db of 600.000 images.

>
> I can add another field to each document, containing a computed
> custom score, then i could sort on that field. But i want to avoid
> this solution at all costs, since it would mean re-indexing all the
> documents each time the collection scores change.

> Or, use indirection - instead of keeping the score, keep the primary
> key of a score table.  Then in a database, where speed won't be the
> issue, perform the look up.   Honestly, if you're only got 300
> categories, you could keep that simple table in memory using less
> space than a small text file.

Again, thats not the space needed to store the lookup table that  
annoys me, its the time it would take to fetch all docs unsorted from  
lucene, then sort using the table.



> I would also like to implement random-sorting. ... Is it a good  
> solution?
> Is there another way to do it?
>
> This really, really, really feels like you're force fitting Lucene to
> do some business logic piece of a larger application.  May I be so
> bold as to ask what's the _actual_ problem you're trying to solve.
> ("I'm trying to make a hole in a piece of oak" as opposed to "What's
> the best way to sharpen a Phillips screwdriver enough to cut wood?")
>
> Keep in mind that the forum is for Lucene, so parts of your questions
> may be answered outside of the forum.

Well, since Sorting/scoring is part of lucene's core api, asking for  
a way to achieve a particular sorting does not seem to me out of the  
scope of this forum/mailing list.

I just need random sorting to give each photographer an equal  
opportunity to sell their work.


Thanks for your answers

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message