lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques du Rand <jacq...@pricecheck.co.za>
Subject Re: CLOSE_WAIT and high search latency
Date Tue, 23 Feb 2016 12:47:49 GMT
I'm not a JVM nor kernel expert so take this with a pinch of salt... my
guess woule be:
 Having more CLOSE_WAIT connections can *probably* affect  you in two ways:

1. Longer GC Cycles pauses (not sure how much of the connection the JVM
handles)
2. Some of the connections might be put on the tcp-backlog

Here is a good article about tcp-tunning
http://engineering.chartbeat.com/2014/01/02/part-1-lessons-learned-tuning-tcp-and-nginx-in-ec2/




On 23 February 2016 at 13:54, Niraj Aswani <niraj.aswani@klevu.com> wrote:

> Hi Jacques,
>
> Thank you for your reply. I'll give these parameters a try and report back.
> Do you think having higher number of CLOSE_WAIT connections affects the
> over all search delivery speed?
>
> Regards,
> Niraj
>
> On Tue, Feb 23, 2016 at 2:57 PM, Jacques du Rand <jacques@pricecheck.co.za
> >
> wrote:
>
> > Try to fiddle with your tcp settings:
> >
> >  /etc/sysctl.conf
> > net.ipv4.tcp_fin_timeout = 30
> > net.ipv4.tcp_keepalive_intvl = 2
> > net.ipv4.tcp_keepalive_probes = 2
> > net.ipv4.tcp_keepalive_time = 180
> >
> > That should help.
> >
> >
> > On 23 February 2016 at 07:13, Niraj Aswani <nirajaswani@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > I am on solr 4.8.1 and running master-slave setup with lots of cores
> > (>3K).
> > > Internally I maintain an instance of HTTPSolrServer for each core that
> is
> > > reused for querying the respective cores. A request is received by an
> > > intermediary tomcat and forwarded to another tomcat running Solr.
> > >
> > > Over the period we see high search latency. Some requests start to take
> > too
> > > long and eventually result into timeouts.
> > >
> > > Investigating this, I see that, over the period, a high number of
> > > CLOSE_WAIT sockets (>3300) are building up. Running `netstat -p` seems
> to
> > > suggest that these sockets were initiated by the intermediary tomcats
> > when
> > > communicating to the Solr.
> > >
> > > Questions are:
> > >
> > > - Why do we see such high number of CLOSE_WAiT sockets? Shouldn't the
> > > HTTPSolrServer take care of closing these connections after
> communicating
> > > with the Solr server?
> > >
> > > - Does the high number of CLOSE_WAIT have anything to do with search
> > > latency?
> > >
> > > Any suggestion on the matter is highly appreciated!
> > >
> > > Regards,
> > > Niraj
> > >
> >
> >
> >
> > --
> > Jacques du Rand
> > Senior R&D  Programmer
> >
> > T: +27214688017
> > F: +27862160617
> > E: jacques@pricecheck.co.za
> > <
> >
> https://mail.naspers.com/owa/redir.aspx?SURL=uQ0aY47BwKkgYjLPs4CuWJgrQlRUZlf73Ere0TGrauf5NrymvPnSCG0AYQBpAGwAdABvADoAagBhAGMAcQB1AGUAcwBAAHAAcgBpAGMAZQBjAGgAZQBjAGsALgBjAG8ALgB6AGEA&URL=mailto%3ajacques%40pricecheck.co.za
> > >
> >
>



-- 
Jacques du Rand
Senior R&D  Programmer

T: +27214688017
F: +27862160617
E: jacques@pricecheck.co.za
<https://mail.naspers.com/owa/redir.aspx?SURL=uQ0aY47BwKkgYjLPs4CuWJgrQlRUZlf73Ere0TGrauf5NrymvPnSCG0AYQBpAGwAdABvADoAagBhAGMAcQB1AGUAcwBAAHAAcgBpAGMAZQBjAGgAZQBjAGsALgBjAG8ALgB6AGEA&URL=mailto%3ajacques%40pricecheck.co.za>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message