lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin de Vries <mar...@downnotifier.com>
Subject SolrCloud constantly crashes after upgrading to Solr 4.7
Date Thu, 06 Mar 2014 22:37:42 GMT
 

Hi, 

We have 5 Solr servers in a Cloud with about 70 cores and 12GB
indexes in total (every core has 2 shards, so it's 6 GB per server).


After upgrade to Solr 4.7 the Solr servers are crashing constantly
(each server about one time per hour). We currently don't have any clue
about the reason. We tried loads of different settings, but nothing
works out. 

When a server crashes the last log item is (most times) a
"Broken pipe" error. The last queries / used cores are completely random
(as far as we can see). 

We are running with the -Xloggc switch and
during a crash it says: 

10838.015: [Full GC
3141724K->3141724K(3522560K), 1.6936710 secs]
10839.710: [Full GC
3141724K->3141724K(3522560K), 1.5682250 secs]
10841.279: [Full GC
3141728K->3141726K(3522560K), 1.5735450 secs]
10842.854: [Full GC
3141727K->3141727K(3522560K), 1.5773380 secs]
10844.433: [Full GC
3141732K->3141687K(3522560K), 1.5696950 secs]
10846.003: [Full GC
3141698K->3141687K(3522560K), 1.5766940 secs]
10847.581: [Full GC
3141695K->3141688K(3522560K), 1.5879360 secs]
10849.170: [Full GC
3141695K->3141691K(3522560K), 1.5698630 secs]
10850.741: [Full GC
3141695K->3141689K(3522560K), 1.5643990 secs]
10852.307: [Full GC
3141693K->3141650K(3522560K), 1.5759150 secs]

We tried to increase the
memory, but that didn't help. We increased the zkClientTimeout to 60
seconds, but that didn't help. 

We made a memory dump with jmap. The
IndexSchema is using 62% of the memory but we don't know if that's a
problem:
https://www.dropbox.com/s/eyom5c48vhl0q9i/Screenshot%202014-03-06%2023.32.41.png
[1] 

Tomorrow we will downgrade each server to Solr 4.6.1, we need to
reindex every core to do that.... unless we have a solution. 

Does
anyone have a clue what the problem can be? 

Thanks! 

Martin 




Links:
------
[1]
https://www.dropbox.com/s/eyom5c48vhl0q9i/Screenshot%202014-03-06%2023.32.41.png

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message