lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Falck" <>
Subject SV: 30 milllion+ docs on a single server
Date Mon, 14 Aug 2006 13:02:54 GMT
You will run into problems with the sorting if you can't hold the fieldcache for long intervals.
I'm working on a system containing 300 million docs. And I ran into sorting issues after only
5 million docs, but then again I can't hold my IndexSearcher open for so long intervals since
I'm dealing with frequently changed data. I had to redo some of the scoring part to just deal
with the boost value to accomplish my sorting. Which however worked nice but I'm looked into
once specific sort.

I'm also pretty interested in how large indexes I will be able to host per server, but I have
already implemented abilities for horizontal scaling I just hope I won't end up with to many



-----Ursprungligt meddelande-----
Från: Mark Miller [] 
Skickat: den 13 augusti 2006 01:45
Ämne: Re: 30 milllion+ docs on a single server

The single server is important because I think it will take a lot of 
work to scale it to multiple servers. The index must allow for close to 
real-time updates and additions. It must also remain searchable at all 
times (other than than during the brief period of single updates and 
additions). If it is easy to scale this to multiple servers please tell 
me how.

- Mark
> Why is a single server so important?  I can scale horizontally much 
> cheaper
> than I scale vertically.
> On 8/11/06, Mark Miller <> wrote:
>> I've made a nice little archive application with lucene. I made it to
>> handle our largest need: 2.5 million docs or so on a single server. Now
>> the powers that be say: lets use it for a 30+ million document archive
>> on a single server! (each doc size maybe 10k small as a 1 or
>> 2k) Please tell me why we are in trouble...please tell me why we are
>> not. I have tested up to 2 million docs without much trouble but 30
>> million...the average search will include a sort on a field as
>> well...can I search 30+ million docs with a sort? Man am I worried about
>> that. Maybe the server will have 8 procs and 12 billion gigs of RAM.
>> Mabye. Even still, Tomcat seems to be able to launch with a max of 1.5
>> or 1.6 gig of Ram in Windows. What do you think? 30 million+ sounds like
>> too much of a load to me for a single server. Not that they care what I
>> think...I only wrote the thing (man I hate my job, offer me a new one :)
>> )...please...comments?
>> Cheers,
>> Miserable Mark
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message