lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy <angelf...@yahoo.com>
Subject Re: Scalability of Lucene indexes
Date Sat, 19 Feb 2005 09:24:52 GMT
Hi Bryan,

How big is your index?

Also what is the advantage of binding a user to a
server? 

Thanks.
Andy

--- Bryan McCormick <bryan.mccormick@cnet.com> wrote:

> Hi chris, 
> 
> I'm responsible for the webshots.com search index
> and we've had very
> good results with lucene. It currently indexes over
> 100 Million
> documents and performs 4 Million searches / day. 
> 
> We initially tested running multiple small copies
> and using a
> MultiSearcher and then merging results as compared
> to running a very
> large single index. We actually found that the
> single large instance
> performed better. To improve load handling we
> clustered multiple
> identical copies together, then session bind a user
> to particular server
> and cache the results, but each server is running a
> single index. 
> 
> Bryan McCormick
> 
> 
> On Fri, 2005-02-18 at 08:01, Chris D wrote: 
> > Hi all, 
> > 
> > I have a question about scaling lucene across a
> cluster, and good ways
> > of breaking up the work.
> > 
> > We have a very large index and searches sometimes
> take more time than
> > they're allowed. What we have been doing is during
> indexing we index
> > into 256 seperate indexes (depending on the
> md5sum) then distribute
> > the indexes to the search machines. So if a
> machine has 128 indexes it
> > would have to do 128 searches. I gave
> parallelMultiSearcher a try and
> > it was significantly slower than simply iterating
> through the indexes
> > one at a time.
> > 
> > Our new plan is to somehow have only one index per
> search machine and
> > a larger main index stored on the master.
> > 
> > What I'm interested to know is whether having one
> extremely large
> > index for the master then splitting the index into
> several smaller
> > indexes (if this is possible) would be better than
> having several
> > smaller indexes and merging them on the search
> machines into one
> > index.
> > 
> > I would also be interested to know how others have
> divided up search
> > work across a cluster.
> > 
> > Thanks,
> > Chris
> > 
> >
>
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> > 
> 
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> 
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message