hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Getting SocketTimeoutException after a very short amount of time
Date Mon, 31 Aug 2015 02:39:21 GMT
I am seeing SocketTimeoutException in the log for my SolrJ program. 
SolrJ uses HttpClient.

I'm starting with the HC list for this problem because that seems like
the most likely place for a problem, but I know that it could be in
SolrJ or my own code.

The index cycle that timed out began at 06:13:30.003.  This starts out
talking to a MySQL database to work out any index changes that need to
be made, and if any are found, proceeds to make them.  The loop failed
with a socket timeout error at 2015-08-30 06:14:00.232, so it looks like
it only waited 30 seconds ... but the SO_TIMEOUT on the httpclient
should have been set to 15 minutes.

Here is the full exception, slightly redacted to obscure my company name:

ERROR - 2015-08-30 06:14:00.232;   977; 2; chain.b: error committing shard 2
com.REDACTED.idxbuild.util.BuildException: shard.b.2.spark2live commit
failed
        at com.REDACTED.idxbuild.solr.Core.commit(Core.java:496)
        at com.REDACTED.idxbuild.index.Shard.commit(Shard.java:454)
        at com.REDACTED.idxbuild.index.Chain$2.run(Chain.java:969)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout
occured while waiting response from server at:
http://idxb3.REDACTED.com:8981/solr
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:570)
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
        at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
        at
org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:523)
        at com.REDACTED.idxbuild.solr.Core.commit(Core.java:484)
        ... 2 more
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
        at
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
        at
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
        at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
        at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
        at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
        at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
        at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
        at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
        at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
        at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
        at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466)
        ... 7 more

Here's my code that creates the two HttpClient objects which are shared
among multiple HttpSolrClient instances.  I create a client with a 15
minute socket timeout for general use, and a client with a two hour
timeout for doing index optimizes:

  synchronized (firstInstance)
  {
      if (firstInstance)
      {
          ModifiableSolrParams params = new ModifiableSolrParams();
          params.add(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, "300");
          params.add(HttpClientUtil.PROP_MAX_CONNECTIONS, "5000");
          params.add(HttpClientUtil.PROP_CONNECTION_TIMEOUT, "15000");
          params.add(HttpClientUtil.PROP_SO_TIMEOUT, "900000");
          httpClient = HttpClientUtil.createClient(params);
          params.clear();
          params.add(HttpClientUtil.PROP_CONNECTION_TIMEOUT, "15000");
          params.add(HttpClientUtil.PROP_SO_TIMEOUT, "7200000");
          optimizeHttpClient = HttpClientUtil.createClient(params);
          firstInstance = false;
      }
  }

httpClient and optimizeHttpClient are static instances of HttpClient.

The code for SolrJ's HttpClientUtil is in Apache SVN.  It is still using
a lot of deprecated methods.  There were a lot of deprecations in
HttpClient 4.3 that have not yet been removed from SolrJ.

Below is the list of jars that I have available to my SolrJ program. 
One of them is the 5.2.1 version of SolrJ.  The solrj dependencies that
are shipped with Solr include HttpClient 4.4.1.  I have upgraded
versions of some components, including HttpClient.  Most of these
dependencies are for SolrJ, but there are some that are only for my program:

commons-dbcp2-2.1.1.jar
commons-io-2.4.jar
commons-pool2-2.4.2.jar
httpclient-4.5.jar
httpcore-4.4.1.jar
httpmime-4.5.jar
javax.mail.jar
jcl-over-slf4j-1.7.12.jar
jrobin-1.5.14.jar
jul-to-slf4j-1.7.12.jar
log4j-1.2.17.jar
mysql-connector-java-5.1.35-bin.jar
noggit-0.7.jar
slf4j-api-1.7.12.jar
slf4j-log4j12-1.7.12.jar
solr-solrj-5.2.1.jar
stax2-api-3.1.1.jar
woodstox-core-asl-4.2.0.jar
zookeeper-3.4.6.jar

Where should I start in the processof getting to the bottom of this
problem?  The first thing I am going to try is creating separate
instances of ModifiableSolrParams for each HttpClient instance.  If
that's the problem, I would expect different symptoms, but it's a
potential code mistake that I did notice.

If you have questions about how HttpClientUtil works, I can attempt to
decipher the code.

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message