lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <s...@elyograg.org>
Subject Re: SolrJ Socket Leak
Date Thu, 13 Feb 2014 21:11:22 GMT
On 2/13/2014 1:38 PM, Jared Rodriguez wrote:
> I am using solr/solrj 4.6.1 along with the apache httpclient 4.3.2 as part
> of a web application which connects to the solr server via solrj
> using CloudSolrServer();  The web application is wired up with Guice, and
> there is a single instance of the CloudSolrServer class used by all inbound
> requests.  All this is running on Amazon.
>
> Basically, everything looks and runs fine for a while, but even with
> moderate concurrency, solrj starts leaving sockets open.  We are handling
> only about 250 connections to the web app per minute and each of these
> issues from 3 - 7 requests to solr.  Over a 30 minute period of this type
> of use, we end up with many 1000s of lingering sockets.  I can see these
> when running netstats
>
> tcp        0      0 ip-10-80-14-26.ec2.in:41098 ip-10-99-145-47.ec2.i:glrpc
> TIME_WAIT
>
> All to the same target host, which is my solr server. There are no other
> pieces of infrastructure on that box, just solr.  Eventually, the server
> just dies as no further sockets can be opened and the opened ones are not
> reused.
>
> The solr server itself is unphased and running like a champ.  Average timer
> per request of 0.126, as seen in the solr web app admin UI query handler
> stats.
>
> Apache httpclient had a bunch of leakage from version 4.2.x that they
> cleaned up and refactored in 4.3.x, which is why I upgraded.  Currently,
> solrj makes use of the old leaky 4.2 classes for establishing connections
> and using a connection pool.

This is something that I can look into.

I have a SolrJ program with SolrJ 4.5.1 and HttpClient 4.3.1 that does 
not leak anything.  I thought it was migrated already to SolrJ 4.6.1, 
but now that I know it's not, I will upgrade SolrJ first and then 
HttpClient, and see whether I have the same problem with either upgrade.

I am using HttpSolrServer, not CloudSolrServer, because the Solr servers 
are not running SolrCloud.  CloudSolrServer ultimately uses 
HttpSolrServer for its communication, so my initial thought is that this 
is not important, but we'll see.

In version 4.7, Solr will include HttpClient 4.3.1.  See SOLR-5590.

https://issues.apache.org/jira/browse/SOLR-5590

A question for committers with a lot of experience: Do we have any tests 
that check for connection leaks?

Thanks,
Shawn


Mime
View raw message