lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: Scaling to large Number of Collections
Date Sun, 31 Aug 2014 16:39:52 GMT
>
> so you might still end up with these out of threads issue again.


You can also generally drop the stack size (Xss) quite a bit to to handle
more threads.

Beyond that, there are some thread pools you can configure. However, until
we fix the distrib deadlock issue, you don't want to drop the container
thread pool too much. There are other control points though.

- Mark
http://about.me/markrmiller


On Sun, Aug 31, 2014 at 11:53 AM, Ramkumar R. Aiyengar <
andyetitmoves@gmail.com> wrote:

> On 31 Aug 2014 13:24, "Mark Miller" <markrmiller@gmail.com> wrote:
> >
> >
> > > On Aug 31, 2014, at 4:04 AM, Christoph Schmidt <
> christoph.schmidt@moresophy.de> wrote:
> > >
> > > we see at least two problems when scaling to large number of
> collections. I would like to ask the community, if they are known and maybe
> already addressed in development:
> > > We have a SolrCloud running with the following numbers:
> > > -          5 Servers (each 24 CPUs, 128 RAM)
> > > -          13.000 Collection with 25.000 SolrCores in the Cloud
> > > The Cloud is working fine, but we see two problems, if we like to scale
> further
> > > 1.       Resource consumption of native system threads
> > > We see that each collection opens at least two threads: one for the
> zookeeper (coreZkRegister-1-thread-5154) and one for the searcher
> (searcherExecutor-28357-thread-1)
> > > We will run in "OutOfMemoryError: unable to create new native thread".
> Maybe the architecture could be changed here to use thread pools?
> > > 2.       The shutdown and the startup of one server in the SolrCloud
> takes 2 hours. So a rolling start is about 10h. For me the problem seems to
> be that leader election is "linear". The Overseer does core per core. The
> organisation of the cloud is not done parallel or distributed. Is this
> already addressed by https://issues.apache.org/jira/browse/SOLR-5473 or is
> there more needed?
> >
> > 2. No, but it should have been fixed by another issue that will be in
> 4.10.
>
> Note however that this fix will result in even more temporary thread usage
> as all leadership elections will happen in parallel, so you might still end
> up with these out of threads issue again.
>
> Quite possibly the out of threads issue is just some system soft limit
> which is kicking in. Linux certainly has a limit you can configure through
> sysctl, your OS, whatever that might be, probably does the same. May be
> worth exploring if you can bump that up.
>
> >
> > - Mark
> > http://about.me/markrmiller

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message