oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantinos Mavrommatis <kmavromma...@celgene.com>
Subject RE: File manager keeps connections to SOLR in CLOSE_WAIT state for hours
Date Wed, 09 Jul 2014 09:21:03 GMT
Hi Lewis,
Thank you for your quick response:
Information about the system:
OS is MacOS Mavericks, 
the SOLR version is 4.6.1 running on Tomcat 7.0.50,
SOLR runs on a single server, no replication at all.
OODT version is 0.6

The interesting thing is that after a day of 'moderate' activity I have now
198 CLOSE_WAIT instances on port 8081 (which is where SOLR is listening) with corresponding
instances in state FIN_WAIT_2 from the client side (i.e. filemanager)
I also have another 104 CLOSE_WAIT instances on port 9000, which is where filemanager listens
to for connections from the compute nodes of the cluster. 
e.g.
java    92065 kmavrommatis  363u  IPv6 0x392c3fa39f6adf0f      0t0  TCP 10.130.0.26:cslistener->cluster:51194
(CLOSE_WAIT)
java    92065 kmavrommatis  364u  IPv6 0x392c3fa3b62759cf      0t0  TCP 10.130.0.26:cslistener->cluster:33123
(CLOSE_WAIT)
java    92065 kmavrommatis  365u  IPv6 0x392c3fa3b6d3314f      0t0  TCP 10.130.0.26:cslistener->cluster:35167
(CLOSE_WAIT)
java    92065 kmavrommatis  366u  IPv6 0x392c3fa3b5faff0f      0t0  TCP 10.130.0.26:cslistener->cluster:46788
(CLOSE_WAIT)
java    92065 kmavrommatis  367u  IPv6 0x392c3fa39f6abd0f      0t0  TCP 10.130.0.26:cslistener->cluster:52658
(CLOSE_WAIT)
java    92065 kmavrommatis  368u  IPv6 0x392c3fa3b6d15e0f      0t0  TCP 10.130.0.26:cslistener->cluster:45768
(CLOSE_WAIT)
java    92065 kmavrommatis  369u  IPv6 0x392c3fa3b60fa58f      0t0  TCP 10.130.0.26:cslistener->cluster:58830
(CLOSE_WAIT)
java    92065 kmavrommatis  370u  IPv6 0x392c3fa39fd849cf      0t0  TCP 10.130.0.26:cslistener->cluster:59112
(CLOSE_WAIT)


I am not sure how to debug this problem, my TCP stack knowledge is limited, any ideas?
Thanks
Konstantinos

> -----Original Message-----
> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> Sent: Wednesday, July 09, 2014 4:55 AM
> To: dev@oodt.apache.org
> Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT state
> for hours
> 
> Hi Konstantinos,
> OK, I was ablel to scope this one and I have a few questions for you.
> 1) Which version of Solr are you using? Is it  3.5, 3.6, 4.0-ALPHA? If
> so please scope this issue [0], the solution would be to upgrade if you
> are not too long ahead with ingestion as fixes in Solr are worth having
> based on recent release cycles.
> 2) How many cores do you have on Solr server? Also what kind of setUp
> do you have? Replication at all?
> In recent versions of Solr 4.X SolrJ clients should now call shutdown()
> on their SolrServer object to let it know they don't want to re-use any
> existing connections anymore, and when Solr internally uses SolrJ to
> talk to other nodes in SolrCloud it should be doing this (as of 4.0-
> ALPHA)so this is why I ask.
> Lewis
> 
> [0] https://issues.apache.org/jira/browse/SOLR-3280
> 
> 
> On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis <
> kmavrommatis@celgene.com> wrote:
> 
> > Hi,
> > I have setup OODT filemanager on port 9000, using SOLR as the
> indexing
> > service on port 8081. They are both setup on the same computer, while
> > crawler runs on a number of different compute nodes spread across the
> > local network and the cloud.
> >
> > When the crawler runs and ingests files I notice that there are
> > several connections that open to solr and remain in CLOSE_WAIT state
> for hours.
> > any idea why this happens? Moving forward I am planning to use
> several
> > hundreds of crawler instances, each running on different computer,
> > that will create thousands of such connections and will probably
> > create problems to the system.
> > Thanks in advance for any help
> > Kostas
> >
> >  $lsof -i :8081
> > COMMAND   PID         USER   FD   TYPE             DEVICE SIZE/OFF
> NODE
> > NAME
> > java    92065 kmavrommatis   75u  IPv6 0x392c3fa3b63b29cf      0t0
> TCP
> > localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT)
> > java    92065 kmavrommatis   76u  IPv6 0x392c3fa3b6dcbbcf      0t0
> TCP
> > localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT)
> > java    92065 kmavrommatis   77u  IPv6 0x392c3fa39fd12e0f      0t0
> TCP
> > localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT)
> > java    92065 kmavrommatis   78u  IPv6 0x392c3fa39fdcdbcf      0t0
> TCP
> > localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT)
> > java    92065 kmavrommatis   79u  IPv6 0x392c3fa3b62cde0f      0t0
> TCP
> > localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT)
> > java    92065 kmavrommatis   80u  IPv6 0x392c3fa39fa2714f      0t0
> TCP
> > localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT)
> > java    92065 kmavrommatis   81u  IPv6 0x392c3fa3b6c32acf      0t0
> TCP
> > localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT)
> > java    92065 kmavrommatis   82u  IPv6 0x392c3fa3b6aa714f      0t0
> TCP
> > localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT)
> >
> >
> > process 92065 is:
> >  /usr/bin/java -Djava.ext.dirs=../lib
> > -Djava.util.logging.config.file=../etc/logging.properties
> > -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum 9000
> >
> > *********************************************************
> > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
> > MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
> > OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> > If the reader is not the intended recipient, or the employee or agent
> > responsible to deliver it to the intended recipient, you are hereby
> > notified that any dissemination, distribution or copying of this
> > communication is strictly prohibited. If you have received this
> > communication in error, please reply to the sender to notify us of
> the
> > error and delete the original message. Thank You.
> >
> 
> 
> 
> --
> *Lewis*

*********************************************************
THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
OR INDIVIDUALS NAMED ABOVE.
If the reader is not the intended recipient, or the
employee or agent responsible to deliver it to the
intended recipient, you are hereby notified that any
dissemination, distribution or copying of this
communication is strictly prohibited. If you have
received this communication in error, please reply to the
sender to notify us of the error and delete the original
message. Thank You.
Mime
View raw message