lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiran Chitturi <kiran.chitt...@lucidworks.com>
Subject Re: SolrJ Socket Leak
Date Mon, 17 Feb 2014 08:03:55 GMT
Jared,

I faced a similar issue when using CloudSolrServer with Solr. As Shawn
pointed out the 'TIME_WAIT' status happens when the connection is closed
by the http client. HTTP client closes connection whenever it thinks the
connection is stale
(https://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html
#d5e405). Even the docs point out the stale connection checking cannot be
all reliable. 

I see two ways to get around this:

	1. Enable 'SO_REUSEADDR'
	2. Disable stale connection checks.

Also by default, when we create CSS it does not explicitly configure any
http client parameters
(https://github.com/apache/lucene-solr/blob/trunk/solr/solrj/src/java/org/a
pache/solr/client/solrj/impl/CloudSolrServer.java#L124). In this case, the
default configuration parameters (max connections, max connections per
host) are used for a http connection. You can explicitly configure these
params when creating CSS using HttpClientUtil:

	ModifiableSolrParams params = new ModifiableSolrParams();
	params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 128);
	params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 32);
	params.set(HttpClientUtil.PROP_FOLLOW_REDIRECTS, false);
	params.set(HttpClientUtil.PROP_CONNECTION_TIMEOUT, 30000);
	httpClient = HttpClientUtil.createClient(params);

	final HttpClient client = HttpClientUtil.createClient(params);
	LBHttpSolrServer lb = new LBHttpSolrServer(client);
	CloudSolrServer server = new CloudSolrServer(zkConnect, lb);


Currently, I am using http client 4.3.2 and building the client when
creating the CSS. I also use 'SO_REUSEADDR' option and I haven't seen the
'TIME_WAIT'  after this (may be because of better handling of stale
connections in 4.3.2 or because of 'SO_REUSEADDR' param enabled). My
current http client code looks like this: (works only with http client
4.3.2)

	HttpClientBuilder httpBuilder = HttpClientBuilder.create();
    
	Builder socketConfig =  SocketConfig.custom();
	socketConfig.setSoReuseAddress(true);
	socketConfig.setSoTimeout(10000);
	httpBuilder.setDefaultSocketConfig(socketConfig.build());
	httpBuilder.setMaxConnTotal(300);
	httpBuilder.setMaxConnPerRoute(100);
    
	httpBuilder.disableRedirectHandling();
	httpBuilder.useSystemProperties();
	LBHttpSolrServer lb = new LBHttpSolrServer(httpClient, parser)
	CloudSolrServer server = new CloudSolrServer(zkConnect, lb);


There should be a way to configure socket reuse with 4.2.3 too. You can
try different configurations. I am surprised you have 'TIME_WAIT'
connections even after 30 minutes because 'TIME_WAIT' connection should be
closed by default in 2 mins by O.S I think.


HTH,

-- 
Kiran Chitturi,


On 2/13/14 12:38 PM, "Jared Rodriguez" <jrodriguez@kitedesk.com> wrote:

>I am using solr/solrj 4.6.1 along with the apache httpclient 4.3.2 as part
>of a web application which connects to the solr server via solrj
>using CloudSolrServer();  The web application is wired up with Guice, and
>there is a single instance of the CloudSolrServer class used by all
>inbound
>requests.  All this is running on Amazon.
>
>Basically, everything looks and runs fine for a while, but even with
>moderate concurrency, solrj starts leaving sockets open.  We are handling
>only about 250 connections to the web app per minute and each of these
>issues from 3 - 7 requests to solr.  Over a 30 minute period of this type
>of use, we end up with many 1000s of lingering sockets.  I can see these
>when running netstats
>
>tcp        0      0 ip-10-80-14-26.ec2.in:41098
>ip-10-99-145-47.ec2.i:glrpc
>TIME_WAIT
>
>All to the same target host, which is my solr server. There are no other
>pieces of infrastructure on that box, just solr.  Eventually, the server
>just dies as no further sockets can be opened and the opened ones are not
>reused.
>
>The solr server itself is unphased and running like a champ.  Average
>timer
>per request of 0.126, as seen in the solr web app admin UI query handler
>stats.
>
>Apache httpclient had a bunch of leakage from version 4.2.x that they
>cleaned up and refactored in 4.3.x, which is why I upgraded.  Currently,
>solrj makes use of the old leaky 4.2 classes for establishing connections
>and using a connection pool.
>
>http://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-4.3.x.t
>xt
>
>
>
>-- 
>Jared Rodriguez


Mime
View raw message