lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christoph Schmidt <christoph.schm...@moresophy.de>
Subject Scaling to large Number of Collections
Date Sun, 31 Aug 2014 08:04:46 GMT
we see at least two problems when scaling to large number of collections. I would like to ask
the community, if they are known and maybe already addressed in development:
We have a SolrCloud running with the following numbers:
 -          5 Servers (each 24 CPUs, 128 RAM)
-          13.000 Collection with 25.000 SolrCores in the Cloud
The Cloud is working fine, but we see two problems, if we like to scale further
1.       Resource consumption of native system threads
We see that each collection opens at least two threads: one for the zookeeper (coreZkRegister-1-thread-5154)
and one for the searcher (searcherExecutor-28357-thread-1)
We will run in "OutOfMemoryError: unable to create new native thread". Maybe the architecture
could be changed here to use thread pools?
2.       The shutdown and the startup of one server in the SolrCloud takes 2 hours. So a rolling
start is about 10h. For me the problem seems to be that leader election is "linear". The Overseer
does core per core. The organisation of the cloud is not done parallel or distributed. Is
this already addressed by https://issues.apache.org/jira/browse/SOLR-5473 or is there more
needed?

Thanks for discussion and help
Christoph
_______________________________________________

Dr. Christoph Schmidt | Geschäftsführer

P +49-89-523041-72
M +49-171-1419367
Skype: cs_moresophy
christoph.schmidt@moresophy.de<mailto:heiko.beier@moresophy.de>
www.moresophy.com<http://www.moresophy.com/>
moresophy GmbH | Fraunhoferstrasse 15 | 82152 München-Martinsried


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message