oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ramirez, Paul M (398J)" <paul.m.rami...@jpl.nasa.gov>
Subject Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to SOLR in CLOSE_WAIT state for hours
Date Thu, 10 Jul 2014 14:05:40 GMT
Konstantinos,

Could you open a Jira issue for this and attach a patch? This would be helpful for the project.


https://issues.apache.org/jira/browse/OODT/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel


Thanks,
Paul Ramirez 

> On Jul 10, 2014, at 5:23 AM, "Konstantinos Mavrommatis" <kmavrommatis@celgene.com>
wrote:
> 
> Hi,
> 
> Following the suggestion at
> 
> http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-avoid-closewait-tcp-connections/
> 
> 
> 
> I modified the code in
> 
> src/main/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrClient.java
> 
> 
> 
> and added the line highlighted in red (method.setRequestHeader(“Connection”,”close”).
> 
> This seems to have resolved the problem at least until now ingestion of few thousand
files have not produced any stale connection.
> 
> Please let me know your thoughts on this solution.
> 
> Best,
> 
> Konstantinos
> 
> 
> 
> ======================================================
> 
> private String doHttp(HttpMethod method) throws Exception {
> 
> 
> 
>                StringBuilder response = new StringBuilder();
> 
>                BufferedReader br = null;
> 
>                try {
> 
> 
> 
>            // send request
> 
>                        HttpClient httpClient = new HttpClient();
> 
>                        // 10 July 2014. attempting to avoid problems with CLOSE_WAIT
connections Based on
> 
>                        //http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-avoid-closewait-tcp-connections/
> 
>                        method.setRequestHeader("Connection", "close");
> 
>            int statusCode = httpClient.executeMethod(method);
> 
> 
> 
>            // read response
> 
>            if (statusCode != HttpStatus.SC_OK) {
> 
> 
> 
>                // still consume the response
> 
>                method.getResponseBodyAsString();
> 
>              throw new CatalogException("HTTP method failed: " + method.getStatusLine());
> 
> 
> 
>            } else {
> 
> 
> 
>                    // read the response body.
> 
>                    br = new BufferedReader(new InputStreamReader(method.getResponseBodyAsStream()));
> 
>        String readLine;
> 
>        while(((readLine = br.readLine()) != null)) {
> 
>          response.append(readLine);
> 
>        }
> 
> 
> 
> =======================================================
> 
> 
> 
> Information about the system:
> 
> OS is MacOS Mavericks,
> 
> SOLR version : 4.6.1 , SOLR runs on a single server, no replication at all.
> 
> Tomcat verision: 7.0.50
> 
> OODT version is 0.6
> 
> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> 
> 
> 
> 
> 
> 
> 
>> -----Original Message-----
> 
>> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> 
>> Sent: Wednesday, July 09, 2014 4:57 AM
> 
>> To: dev@oodt.apache.org
> 
>> Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT state
> 
>> for hours
> 
> 
>> It's just occoured to me that everything in
> 
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog (well,
> 
>> specifically
> 
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient) is done through
> 
>> HTTPClient so all of the above may not be relevant.
> 
>> Can you please scope all the same?
> 
>> Thanks
> 
>> Lewis
> 
> 
> 
>> On Tue, Jul 8, 2014 at 10:54 PM, Lewis John Mcgibbney <
> 
>> lewis.mcgibbney@gmail.com<mailto:lewis.mcgibbney@gmail.com>> wrote:
> 
> 
>>> Hi Konstantinos,
> 
>>> OK, I was ablel to scope this one and I have a few questions for you.
> 
>>> 1) Which version of Solr are you using? Is it  3.5, 3.6, 4.0-ALPHA?
> 
>> If
> 
>>> so please scope this issue [0], the solution would be to upgrade if
> 
>>> you are not too long ahead with ingestion as fixes in Solr are worth
> 
>>> having based on recent release cycles.
> 
>>> 2) How many cores do you have on Solr server? Also what kind of setUp
> 
>>> do you have? Replication at all?
> 
>>> In recent versions of Solr 4.X SolrJ clients should now call
> 
>>> shutdown() on their SolrServer object to let it know they don't want
> 
>>> to re-use any existing connections anymore, and when Solr internally
> 
>>> uses SolrJ to talk to other nodes in SolrCloud it should be doing
> 
>> this
> 
>>> (as of 4.0-ALPHA)so this is why I ask.
> 
>>> Lewis
> 
> 
>>> [0] https://issues.apache.org/jira/browse/SOLR-3280
> 
> 
> 
>>> On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis <
> 
>>> kmavrommatis@celgene.com<mailto:kmavrommatis@celgene.com>> wrote:
> 
> 
>>>> Hi,
> 
>>>> I have setup OODT filemanager on port 9000, using SOLR as the
> 
>>>> indexing service on port 8081. They are both setup on the same
> 
>>>> computer, while crawler runs on a number of different compute nodes
> 
>>>> spread across the local network and the cloud.
> 
> 
>>>> When the crawler runs and ingests files I notice that there are
> 
>>>> several connections that open to solr and remain in CLOSE_WAIT state
> 
>> for hours.
> 
>>>> any idea why this happens? Moving forward I am planning to use
> 
>>>> several hundreds of crawler instances, each running on different
> 
>>>> computer, that will create thousands of such connections and will
> 
>>>> probably create problems to the system.
> 
>>>> Thanks in advance for any help
> 
>>>> Kostas
> 
> 
>>>> $lsof -i :8081
> 
>>>> COMMAND   PID         USER   FD   TYPE             DEVICE SIZE/OFF
> 
>> NODE
> 
>>>> NAME
> 
>>>> java    92065 kmavrommatis   75u  IPv6 0x392c3fa3b63b29cf      0t0
> 
>> TCP
> 
>>>> localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT)
> 
>>>> java    92065 kmavrommatis   76u  IPv6 0x392c3fa3b6dcbbcf      0t0
> 
>> TCP
> 
>>>> localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT)
> 
>>>> java    92065 kmavrommatis   77u  IPv6 0x392c3fa39fd12e0f      0t0
> 
>> TCP
> 
>>>> localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT)
> 
>>>> java    92065 kmavrommatis   78u  IPv6 0x392c3fa39fdcdbcf      0t0
> 
>> TCP
> 
>>>> localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT)
> 
>>>> java    92065 kmavrommatis   79u  IPv6 0x392c3fa3b62cde0f      0t0
> 
>> TCP
> 
>>>> localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT)
> 
>>>> java    92065 kmavrommatis   80u  IPv6 0x392c3fa39fa2714f      0t0
> 
>> TCP
> 
>>>> localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT)
> 
>>>> java    92065 kmavrommatis   81u  IPv6 0x392c3fa3b6c32acf      0t0
> 
>> TCP
> 
>>>> localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT)
> 
>>>> java    92065 kmavrommatis   82u  IPv6 0x392c3fa3b6aa714f      0t0
> 
>> TCP
> 
>>>> localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT)
> 
> 
> 
>>>> process 92065 is:
> 
>>>> /usr/bin/java -Djava.ext.dirs=../lib
> 
>>>> -Djava.util.logging.config.file=../etc/logging.properties
> 
>>>> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
> 
>>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum 9000
> 
> 
>>>> *********************************************************
> 
>>>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
> 
>>>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
> 
>>>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> 
>>>> If the reader is not the intended recipient, or the employee or
> 
>> agent
> 
>>>> responsible to deliver it to the intended recipient, you are hereby
> 
>>>> notified that any dissemination, distribution or copying of this
> 
>>>> communication is strictly prohibited. If you have received this
> 
>>>> communication in error, please reply to the sender to notify us of
> 
>>>> the error and delete the original message. Thank You.
> 
> 
> 
> 
> 
>>> --
> 
>>> *Lewis*
> 
> 
> 
> 
> 
>> --
> 
>> *Lewis*
> 
> *********************************************************
> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
> CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
> INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
> OR INDIVIDUALS NAMED ABOVE.
> If the reader is not the intended recipient, or the
> employee or agent responsible to deliver it to the
> intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this
> communication is strictly prohibited. If you have
> received this communication in error, please reply to the
> sender to notify us of the error and delete the original
> message. Thank You.

Mime
View raw message