lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: Clustering Lucene with 40 Servers
Date Tue, 02 Jan 2007 09:24:27 GMT
Yes, if you think about how blogs and Technorati work (new blog post -> ping -> ping
server -> technorati -> index ==> searchable blog post), Adam is correct.  Since
Doug's implementation we (Technorati) have changed our clusters a LOT (think major rewrites).
 I can't talk about the details, obviously, but the first paragraph in Mark's email sounds
like what we have.

Biggy will want to provide more details about how they want things to work (e.g. how quickly
do changes need to be visible to searchers?  Is this a write + read system, or are there deletes?
...) before others can make useful suggestions.

Otis

----- Original Message ----
From: Adam Fleming <aflem26@hotmail.com>
To: java-user@lucene.apache.org
Sent: Thursday, December 28, 2006 2:33:37 AM
Subject: RE: Clustering Lucene with 40 Servers


Hello, 

I saw that Doug Cutting had an interesting solution for his Technorati website: 
http://www.mail-archive.com/lucene-user@jakarta.apache.org/msg12709.html

It sounds like it's a single-writer, many readers type of system, but quite robust and efficient.

Cheers, 

Adam 



----------------------------------------
> Date: Wed, 27 Dec 2006 10:46:47 -0800
> From: sharrissf@gmail.com
> To: java-user@lucene.apache.org
> Subject: Re: Clustering Lucene with 40 Servers
> 
> Some quick questions/points:
> 
> What is the update rate?
> 
> The number of nodes you described is no problem, the query rate would be
> no problem too (because they use read locks and act independently).
> 
> Do all nodes do updates or just 1? How often do these updates occur?
> 
> Probably best thing to do is give it a try and run a few tests.
> 
> Cheers,
> Steve
> 
> 
> On 12/27/06, Biggy <biggy97@web.de> wrote:
> >
> > I'm currently investigating the best ways of clustering Lucene.
> > I've heard of both Solr, Terracotta but do not know how well they scale.
> > Their examples talk of a 4 node cluster. This is way too small for my needs.
> >
> > I have 30x JVMs each handling 3 requests/sec and each having their own
> > Lucene index. The index changes are propagated to the cluster members using
> > JGroups messages. This solution has more than reached its limit as JGroups
> > has become unstable and a source of many JVMs crashes. Based on current
> > traffic trends I anticipate needing to upgrade to 40x + JVMS very soon.
> >
> > Can anybody suggest a way to effectivily cluster / replicate document
> > changes ?
> >
> > P.S.:
> > JMS is not a possible solution as this was the prior JGroups solution. We
> > had too many memory problems/queue full etc crashing the servers thereafter.
> >
> > --
> > View this message in context: http://www.nabble.com/Clustering-Lucene-with-40-Servers-tf2886546.html#a8064135
> > Sent from the Lucene - Java Users mailing list archive at Nabble.com.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 

_________________________________________________________________
Fixing up the home? Live Search can help.
http://imagine-windowslive.com/search/kits/default.aspx?kit=improve&locale=en-US&source=wlmemailtaglinenov06
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org





---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message