lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Issue serving concurrent requests to SOLR on PROD
Date Tue, 19 May 2015 12:46:23 GMT
On 5/19/2015 1:51 AM, Jani, Vrushank wrote:
> We have production SOLR deployed on AWS Cloud. We have currently 4 live SOLR servers
running on m3xlarge EC2 server instances behind ELB (Elastic Load Balancer) on AWS cloud.
We run Apache SOLR in Tomcat container which is sitting behind Apache httpd. Apache httpd
is using prefork mpm and the request flows from ELB to Apache Httpd Server to Tomcat (via
AJP).
> 
> Last few days, we are seeing increase in the requests around 20000 requests minute hitting
the LB. In effect we see ELB Surge Queue Length continuously being around 100.
> Surge Queue Length: represents the total number of request pending submission to the
instances, queued by the load balancer;
> 
> This is causing latencies and time outs from Client applications. Our first reaction
was that we don't have enough max connections set either in HTTPD or Tomcat. What we saw,
the servers are very lightly loaded with very low CPU and memory utilisation.  Apache preform
settings are as below on each servers with keep-alive turned off.
> 
> <IfModule prefork.c>
> StartServers 8
> MinSpareServers 5
> MaxSpareServers 20
> ServerLimit 256
> MaxClients 256
> MaxRequestsPerChild 4000
> </IfModule>
> 
> 
> Tomcat server.xml has following settings.
> 
> <Connector port="8080" protocol="AJP/1.3" address="127.0.0.1" maxThreads="500" connectionTimeout="60000"/>
> For HTTPD – we see that there are lots of TIME_WAIT connections Apache port around
7000+ but ESTABLISHED connections are around 20.
> For Tomact – we see  about 60 ESTABLISHED connections  on tomcat AJP port.
> 
> So the servers and connections doesn't look like fully utilised to the capacity. There
is no visible stress anywhere.  However we still get requests being queued up on LB because
they can not be served from underlying servers.
> 
> Can you please help me resolving this issue? Can you see any apparent problem here? Am
I missing any configuration or settings for SOLR?

I'm curious about why you have Apache sitting in front of Tomcat.  About
the only reason I can think of to require that step is that you are
using it to require authentication or to deny access to things like the
admin UI.   If you are not doing anything in Apache other than proxying
the traffic, then drop the middleman and use the container directly with
its own HTTP connector.  Or even better, use the Jetty included with Solr.

You should set maxThreads to 10000 in your Tomcat configuration,
effectively removing the limit.  Solr is a multi-threaded Java servlet,
with background threads as well as request-based threads.  Tomcat
requires threads for handling connections, but Solr also requires
threads for its own operation.  The maxThreads limit counts *all* of
those threads, not just the Tomcat threads.

Thanks,
Shawn


Mime
View raw message