lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sundar shankar <sunaish_3...@hotmail.com>
Subject RE: Question on how index works - runs out of disk space!
Date Wed, 10 Sep 2008 21:51:36 GMT
OPtimize solved it . Thanks Jason. I am surprised on why solr does this?



> Date: Wed, 10 Sep 2008 14:37:11 -0400
> From: jrennie@gmail.com
> To: solr-user@lucene.apache.org
> Subject: Re: Question on how index works - runs out of disk space!
> 
> Have you tried performing an "optimize"?  Solr doesn't seem to fully
> integrate all updates into a single index until an optimize is performed.
> 
> Jason
> 
> On Wed, Sep 10, 2008 at 1:05 PM, sundar shankar <sunaish_3000@hotmail.com>wrote:
> 
> > Hi All,
> >          We have a cluster of 4 servers for the application and Just one
> > server for Solr. We have just about 2 million docs to index and we never
> > bothered to make the solr environment clustered as Solr was delivering
> > performance with the current setup itself. Offlate we just discovered a
> > problem and I am not sure what would be the right way to go about this.
> >
> > We have a cron that runs from the application that does a nightly index of
> > data added from another enterprise application. The index job, indexes all
> > courses, be it already indexed or not, and re indexes them. We observed that
> > the job was starting up on all 4 servers at about the same time. All 4
> > servers point to the same Solr box and the same data is apparently added to
> > the solr box 4 times. There is an update command for every 10,000 data
> > fetched from the database and an commit at the end of the full job.
> >
> > The surprising thing that I noticed was that even though there is a primary
> > key defined in the solr schema, the size of the data(folder) seems to
> > incrementaly increase and is causing the solr server to run out of disk
> > space. I have recently upgraded to the 1.3 version about a month back and I
> > guess the problems might be something that is occuring after that update.
> >
> > The index size of a about millions docs on a clustered dev used to be about
> > 520 megs and is about that much the first time index all the courses. The
> > current size of the same number docs (got from stats page) is 6.5 gigs.
> >
> > Am not sure what has changed and if I there any config change that I could
> > use. The write lock is disabled on dev with lock-type = single. I am not
> > sure if this matters.
> >
> > -Sundar
> >
> > _________________________________________________________________
> > Searching for the best deals on travel? Visit MSN Travel.
> > http://in.msn.com/coxandkings
> >
> 
> 
> 
> -- 
> Jason Rennie
> Head of Machine Learning Technologies, StyleFeeder
> http://www.stylefeeder.com/
> Samantha's blog & pictures: http://samanthalyrarennie.blogspot.com/

_________________________________________________________________
Searching for the best deals on travel? Visit MSN Travel.
http://in.msn.com/coxandkings
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message