lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GASPARD Joel <Joel.GASP...@cegedim.com>
Subject RE: Indexing problems
Date Thu, 31 Jan 2013 15:17:30 GMT
Hello Erick,

Thanks for your answer.

After reading previous subjects on the user list, we had already tried to change the parameters
we mentioned.

- concurrent warming searchers : we have set the maxWarmingSearchers attribute to 2 
<maxWarmingSearchers>2</maxWarmingSearchers>

- we have tried 32 and 64 for the ramBufferSizeMB attribute

- there is no other load on the Solr server, or search when we index

- the autocommit is defined with openSearcher=false, maxTime=600000ms, maxDocs=6000 - the
autoSoftCommit is defined with maxTime=1000
We have already tried to change the softcommit and the commit parameters in several ways.
We have also tried to commit on the client size.
Ok I try to commit more often.

- we have used cache sizes defined in the example : size=512

The documents size is not too big, I think : 1 million documents produce a 6Go index.

Thanks for your answer on multiple collections. I thought multiple collections should have
the same schema in Zk after reading a wiki page : 
http://wiki.apache.org/solr/NewSolrCloudDesign : "The entire cluster must have a single schema
and solrconfig"
Maybe is this page deprecated ?
I also thought that because OOM errors occur only when we index a second collection. There
is no problem when indexing a single collection.

Going with 4.1 would not be easy for now... We'll think about it.

Thanks.

Joel


-----Message d'origine-----
De : Erick Erickson [mailto:erickerickson@gmail.com] 
Envoyé : jeudi 31 janvier 2013 14:00
À : solr-user@lucene.apache.org
Objet : Re: Indexing problems

I'm really surprised you're hitting OOM errors, I suspect you have something else pathological
in your system. So, I'd start checking things like
- how many concurrent warming searchers you allow
- How big your indexing RAM is set to (we find very little gain over 128M BTW).
- Other load on your Solr server. Are you, for instance, searching on it too?
- what your autocommit characterstics are (think about autocommitting fairly often with openSearcher=false).
- have you defined huge caches?
- .....

How big are these documents anyway? With 12G of ram, they'd have to be absolutely _huge_ to
matter much.

Multiple collections should work fine in ZK. I really think you have some innocent-looking
configuration setting thats bollixing you up, this is not expected behavior.

If at all possible, I'd also go with 4.1. I don't really think it's relevant to your situation,
but there have been a lot of improvements in the code....

Best
Erick

Mime
View raw message