hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: Getting SocketTimeoutException after a very short amount of time
Date Mon, 31 Aug 2015 09:02:06 GMT
On Sun, 2015-08-30 at 20:39 -0600, Shawn Heisey wrote:
> I am seeing SocketTimeoutException in the log for my SolrJ program. 
> SolrJ uses HttpClient.
> 
> I'm starting with the HC list for this problem because that seems like
> the most likely place for a problem, but I know that it could be in
> SolrJ or my own code.
> 
> The index cycle that timed out began at 06:13:30.003.  This starts out
> talking to a MySQL database to work out any index changes that need to
> be made, and if any are found, proceeds to make them.  The loop failed
> with a socket timeout error at 2015-08-30 06:14:00.232, so it looks like
> it only waited 30 seconds ... but the SO_TIMEOUT on the httpclient
> should have been set to 15 minutes.
> 
> Here is the full exception, slightly redacted to obscure my company name:
> 
> ERROR - 2015-08-30 06:14:00.232;   977; 2; chain.b: error committing shard 2
> com.REDACTED.idxbuild.util.BuildException: shard.b.2.spark2live commit
> failed
>         at com.REDACTED.idxbuild.solr.Core.commit(Core.java:496)
>         at com.REDACTED.idxbuild.index.Shard.commit(Shard.java:454)
>         at com.REDACTED.idxbuild.index.Chain$2.run(Chain.java:969)
> Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout
> occured while waiting response from server at:
> http://idxb3.REDACTED.com:8981/solr
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:570)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
>         at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
>         at
> org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:523)
>         at com.REDACTED.idxbuild.solr.Core.commit(Core.java:484)
>         ... 2 more
> Caused by: java.net.SocketTimeoutException: Read timed out
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.read(SocketInputStream.java:152)
>         at java.net.SocketInputStream.read(SocketInputStream.java:122)
>         at
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
>         at
> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
>         at
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
>         at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
>         at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>         at
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
>         at
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
>         at
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
>         at
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
>         at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
>         at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
>         at
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
>         at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
>         at
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
>         at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>         at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
>         at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466)
>         ... 7 more
> 
> Here's my code that creates the two HttpClient objects which are shared
> among multiple HttpSolrClient instances.  I create a client with a 15
> minute socket timeout for general use, and a client with a two hour
> timeout for doing index optimizes:
> 
>   synchronized (firstInstance)
>   {
>       if (firstInstance)
>       {
>           ModifiableSolrParams params = new ModifiableSolrParams();
>           params.add(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, "300");
>           params.add(HttpClientUtil.PROP_MAX_CONNECTIONS, "5000");
>           params.add(HttpClientUtil.PROP_CONNECTION_TIMEOUT, "15000");
>           params.add(HttpClientUtil.PROP_SO_TIMEOUT, "900000");
>           httpClient = HttpClientUtil.createClient(params);
>           params.clear();
>           params.add(HttpClientUtil.PROP_CONNECTION_TIMEOUT, "15000");
>           params.add(HttpClientUtil.PROP_SO_TIMEOUT, "7200000");
>           optimizeHttpClient = HttpClientUtil.createClient(params);
>           firstInstance = false;
>       }
>   }
> 
> httpClient and optimizeHttpClient are static instances of HttpClient.
> 
> The code for SolrJ's HttpClientUtil is in Apache SVN.  It is still using
> a lot of deprecated methods.  There were a lot of deprecations in
> HttpClient 4.3 that have not yet been removed from SolrJ.
> 
> Below is the list of jars that I have available to my SolrJ program. 
> One of them is the 5.2.1 version of SolrJ.  The solrj dependencies that
> are shipped with Solr include HttpClient 4.4.1.  I have upgraded
> versions of some components, including HttpClient.  Most of these
> dependencies are for SolrJ, but there are some that are only for my program:
> 
> commons-dbcp2-2.1.1.jar
> commons-io-2.4.jar
> commons-pool2-2.4.2.jar
> httpclient-4.5.jar
> httpcore-4.4.1.jar
> httpmime-4.5.jar
> javax.mail.jar
> jcl-over-slf4j-1.7.12.jar
> jrobin-1.5.14.jar
> jul-to-slf4j-1.7.12.jar
> log4j-1.2.17.jar
> mysql-connector-java-5.1.35-bin.jar
> noggit-0.7.jar
> slf4j-api-1.7.12.jar
> slf4j-log4j12-1.7.12.jar
> solr-solrj-5.2.1.jar
> stax2-api-3.1.1.jar
> woodstox-core-asl-4.2.0.jar
> zookeeper-3.4.6.jar
> 
> Where should I start in the processof getting to the bottom of this
> problem?  The first thing I am going to try is creating separate
> instances of ModifiableSolrParams for each HttpClient instance.  If
> that's the problem, I would expect different symptoms, but it's a
> potential code mistake that I did notice.
> 
> If you have questions about how HttpClientUtil works, I can attempt to
> decipher the code.
> 
> Thanks,
> Shawn
> 

Shawn,

The socket timeout in blocking i/o is handled by the JRE. We have no
control over it. 
--- 
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
---
There likely to be something that sets SO_TIMEOUT to a lower value. You
should be able to trace it by placing a breakpoint or good ol'
System.out.println at this line:
http://hc.apache.org/httpcomponents-core-4.4.x/httpcore/xref/org/apache/http/impl/BHttpConnectionBase.html#277

Naturally it would make things easier if Solr used no deprecated
methods.

Oleg 




---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message