lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kent Mu <solr.st...@gmail.com>
Subject Re: sorlcloud connection issue
Date Sat, 09 Jul 2016 09:58:54 GMT
Dear Mr. Mangar!
Thanks for your reply. we have 3 cores with 5 shards, and each shard with
one leader and one replication. now the data account goes to be 100
million, and it is still growing up.I see no connection timeout. actually
we have schedule job to commit the data every 2 hours. so the action to
commit is not that frequent, and no auto-commit. the write operation seems
to be OK.

I just see the reply by Mr. Heisey. He finds "jboss running info in java
core". and he says that we need to raise the maxThread in JBOSS, which is
200 as default. so I decide to config it on next week, and to see if it
will be OK.

Thanks again!

2016-07-07 16:08 GMT+08:00 Shalin Shekhar Mangar <shalinmangar@gmail.com>:

> Hi Kent,
>
> There is no point sending multiple emails for the same subject. It
> distracts people from the other messages, distributes the conversation and
> discourages people from helping you.
>
> Please provide more details about your cluster.
> 1. How many nodes?
> 2. How many collections?
> 3. How many shards?
> 4. What is the replication factor?
> 5. Are there connection and read timeouts specified on both client and
> server? If yes, what are the values?
> 6. How often do you commit? Is there auto-commit or explicit commits from
> clients?
> 7. What else is happening on your cluster? What is the write and query
> load?
>
> The important thing is to figure out why these pools are getting exhausted.
> 10000 max connections and 500 connections per host is already pretty high
> so there is an underlying cause for such stuck requests.
>
> On Wed, Jul 6, 2016 at 4:56 PM, Kent Mu <solr.study@gmail.com> wrote:
>
> > Hi friends!
> > *solr version: 4.9.0*
> >
> > I came across a problem when use solrcloud, it becomes dead lock, we got
> > the java core log, it looks like the http connection pool is exhausted
> and
> > most threads are waiting to get a free connection..
> >
> > I posted the problem in JIRA, the link is
> > https://issues.apache.org/jira/browse/SOLR-9253
> > I have increased http connection defaults for the SolrJ client, and also
> > configed the connection defaults in solr.xml for all shard servers as
> > below.
> >
> > <shardHandlerFactory name="shardHandlerFactory"
> > class="HttpShardHandlerFactory">
> >     <int name="socketTimeout">60000</int>
> >     <int name="connTimeout">30000</int>
> >     <int name="maxConnections">10000</int>
> >     <int name="maxConnectionsPerHost">500</int>
> > </shardHandlerFactory>
> >
> > *besides, we use the singleton pattern to connect solrlcoud.*
> >
> > public synchronized static CloudSolrServer getSolrCloudReadServer() {
> >         if (reviewSolrCloudServer == null) {
> >          ModifiableSolrParams params = new ModifiableSolrParams();
> >          params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 2000);
> >          params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 500);
> >          HttpClient client = HttpClientUtil.createClient(params);
> >          LBHttpSolrServer lbServer = new LBHttpSolrServer(client);
> >
> >
> >
> lbServer.setConnectionTimeout(ReviewProperties.getCloudConnectionTimeOut());
> >            lbServer.setSoTimeout(ReviewProperties.getCloudSoTimeOut());
> >
> >          reviewSolrCloudServer = new
> > CloudSolrServer(ReviewProperties.getZkHost(),lbServer);
> >          reviewSolrCloudServer.setDefaultCollection("commodityReview");
> >
> >
> >
> reviewSolrCloudServer.setZkClientTimeout(ReviewProperties.getZkClientTimeout());
> >
> >
> >
> reviewSolrCloudServer.setZkConnectTimeout(ReviewProperties.getZkConnectTimeout());
> >          reviewSolrCloudServer.connect();
> >         }
> >         return reviewSolrCloudServer;
> >     }
> >
> > *the java stack as below*
> >
> > "httpShardExecutor-3-thread-541" prio=10 tid=0x00007f7b1c02b000
> nid=0x20af
> > waiting on condition [0x00007f79fd490000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x0000000605710068> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> > at
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> > at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133)
> > at
> >
> >
> org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282)
> > at
> >
> org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
> > at
> >
> >
> org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177)
> > at
> >
> >
> org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170)
> > at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102)
> > at
> >
> >
> org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:208)
> > at
> >
> >
> org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)
> > at
> >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:422)
> > at
> >
> >
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
> > at
> >
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> > at
> >
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
> > at
> >
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
> > at
> >
> >
> org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:205)
> > at
> >
> >
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162)
> > at
> >
> >
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:724)
> >
> > please help me, look forward to your reply. Thanks a lot!
> >
> > Best Regards!
> > Kent
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message