lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Burton-West, Tom" <tburt...@umich.edu>
Subject RE: CommonGrams indexing very slow!
Date Wed, 27 Apr 2011 17:09:44 GMT
Hi Salman,

Sounds like somehow you are triggering merges or optimizes.  What is your mergeFactor?  

Have you turned on the IndexWriter log?

In solrconfig.xml
<infoStream file="${solr.indexwriter.log.dir}">true</infoStream>

 In our case we feed the directory name as a Java property in our java startup script , but
you can also hard code where you want the log written like in the current example Solr config:

<infoStream file="INFOSTREAM.txt">false</infoStream>

That should provide some clues.  For example you can see how many segments of each level there
are just before you do the commit that triggers the problem.   My first guess is that you
have enough segments so that adding the documents and committing triggers a cascading merge.
(But this is a WAG without seeing what's in your indexwriter log)

Can you also send your solrconfig so we can see your mergeFactor and ramBufferSizeMB settings?

Tom

> > All,
> >
> > We have created index with CommonGrams and the final size is around
> 370GB.
> > Everything is working fine but now when we add more documents into index
> it
> > takes forever (almost 12 hours)...seems to change all the segments file
> in a
> > commit.
> >
> > The same commit used to take few mins with normal index.
> >
> > Any idea whats going on?
> >
> > --
> > Regards,
> >
> > Salman Akram
> > Principal Software Engineer - Tech Lead
> > NorthBay Solutions
> > 410-G4 Johar Town, Lahore
> > Off: +92-42-35290152
> >
> > Cell: +92-321-4391210 -- +92-300-4009941
> >
>



-- 
Regards,

Salman Akram

Mime
View raw message