lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: "Natural sorting" of documents in a Lucene index - possible?
Date Mon, 16 Aug 2010 20:01:45 GMT
Let's back up a minute. The number of matched records is not
important when sorting, what's important is the number of unique
terms in the field being sorted. Do you have any figures on that? One
very common sorting issue is sorting on very fine date time resolutions,
although your examples don't include that...

Now, cache loading is an issue. The very first time you sort on a field,
all the values are read into a cache. Is it possible this is the source
of your problems? You can cure this with warmup queries. The take-away
is that measuring the response time for the first sorted query is
very misleading.

Although if you're sorting on many, many, many email addresses,
that could be "interesting".

The comment "returning 10,000,000 documents" is, I hope, a
misstatement. If you're trying to *return* 10M docs sorting
is irrelevant compared to assembling that many documents. If
you're trying to return the first 100 of 10M documents, it should
work.

Overall, we need more details on what you're sorting and what
you're trying to return as well as how you're measuring before
we can say much....

Along with how much memory you're giving your JVM to work with,
what "exploding" means. Are you CPU bound? IO bound? Swapping?
You need to characterize what is going wrong before worrying about
solutions......

Best
Erick

On Mon, Aug 16, 2010 at 3:08 PM, Michel Nadeau <akaris@gmail.com> wrote:

> Hi,
>
> we are building an application using Lucene and we have HUGE data sets (our
> index contains millions and millions and millions of documents), which
> obviously cause us very important problems when sorting. In fact, we
> disabled sorting completely because the servers were just exploding when
> trying to sort results in RAM. The users using the system can search for
> whatever they want, so we never know how many results will be returned - a
> search can return 10 documents (no problem with sorting) or 10,000,000
> (huge
> sorting problems).
>
> I woke up this morning and had a flash : is it possible with Lucene to have
> a "natural sorting" of documents? For example, let's say I have 3 columns I
> want to be able to sort by : first name, last name, email; I would have 3
> different indexes with the very same data but with a different primary key
> for sorting. I know it's far fetched, and I have never seen anything like
> that since I use Lucene, but we're just desperate... how people do to have
> huge data sets, a lot of users, and sort!?
>
> Thanks,
>
> Mike
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message