Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Wed, 18 May 2016 14:20:12 +0000 (UTC)
From: "Peter Horvath (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: <JIRA.12970778.1463581167000.224086.1463581212802@Atlassian.JIRA>
In-Reply-To: <JIRA.12970778.1463581167000@Atlassian.JIRA>
References: <JIRA.12970778.1463581167000@Atlassian.JIRA> <JIRA.12970778.1463581167067@arcas>
Subject: [jira] [Created] (SOLR-9129) Solr Cloud hangs when creating large
 number of collections and node fails to recover after restart
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 18 May 2016 14:20:19 -0000

Peter Horvath created SOLR-9129:
-----------------------------------

             Summary: Solr Cloud hangs when creating large number of collec=
tions and node fails to recover after restart
                 Key: SOLR-9129
                 URL: https://issues.apache.org/jira/browse/SOLR-9129
             Project: Solr
          Issue Type: Bug
          Components: Server
    Affects Versions: 6.0
         Environment: OS: GNU Linux, kernel 4.4.0-22 on x86_64 (Ubuntu Linu=
x 16.04 LTS (64-bit))
RAM: 16 GB
CPU: Intel Core i7-4720HQ CPU @ 2.60GHz =C3=97 8
Java version: Oracle JDK 1.8.0_92 (x64) build 1.8.0_92-b14 Java HotSpot(TM)=
 64-Bit Server VM (build 25.92-b14, mixed mode)
            Reporter: Peter Horvath


I attempted to benchmark SolrCloud to see how well it would work with some =
sample data set of ours.=20
I wanted to create about 2500 empty collections first to see how that would=
 scale.

Unfortunately, the test was not successful. Solr started failing after crea=
ting around 2000 collections and the cluster has failed to recover after a =
complete restart, which is quite concerning to me.=20

I based my environment on the cloud example (I use the same config set as t=
he gettingstarted example collection etc); so I have the vanilla install an=
d used the following commands to bring the nodes online:

.../solr/6.0.0/bin/solr start -m 2g -cloud -p 8983 -s
".../solr/6.0.0/example/cloud/node1/solr"
.../solr/6.0.0/bin/solr start -m 2g -cloud -p 7574 -s
".../solr/6.0.0/example/cloud/node2/solr" -z localhost:9983
.../solr/6.0.0/bin/solr start -m 2g -cloud -p 8984 -s
".../solr/6.0.0/example/cloud/node3/solr" -z localhost:9983
.../solr/6.0.0/bin/solr start -m 2g -cloud -p 7575 -s
".../solr/6.0.0/example/cloud/node4/solr" -z localhost:9983

After about 2000 collections were created, SolR got hung; REST requests sta=
rted failing. I found the following entry in the logs, wihch I could relate=
 to the failed REST request. For further logs, please see the attachment of=
 this issue.=20

null:org.apache.solr.common.SolrException: Could not fully create collectio=
n: FOOBAR
=09at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(Colle=
ctionsHandler.java:266)
=09at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(Co=
llectionsHandler.java:197)
=09at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandl=
erBase.java:155)
=09at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.=
java:658)
=09at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)
=09at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilte=
r.java:229)
=09at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilte=
r.java:184)
=09at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet=
Handler.java:1668)
=09at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java=
:581)
=09at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j=
ava:143)
=09at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.jav=
a:548)
=09at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandl=
er.java:226)
=09at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandl=
er.java:1160)
=09at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:=
511)
=09at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandle=
r.java:185)
=09at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle=
r.java:1092)
=09at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j=
ava:141)
=09at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Cont=
extHandlerCollection.java:213)
=09at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerColl=
ection.java:119)
=09at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper=
.java:134)
=09at org.eclipse.jetty.server.Server.handle(Server.java:518)
=09at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
=09at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.jav=
a:244)
=09at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(Abstra=
ctConnection.java:273)
=09at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
=09at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoin=
t.java:93)
=09at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceA=
ndRun(ExecuteProduceConsume.java:246)
=09at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(Exec=
uteProduceConsume.java:156)
=09at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPoo=
l.java:654)
=09at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool=
.java:572)
=09at java.lang.Thread.run(Thread.java:745)

For further logs, please see the attachment of this issue.=20

After the Solr instance affected has failed to recover, I decided to restar=
t the whole cluster (using the official solr stop-start commands). Unfortun=
ately, after this, at least one node remained spinning in ZooKeeper logic, =
creating more than four thousand (!!) threads. =20


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org