oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Mattmann <chris.mattm...@gmail.com>
Subject Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to SOLR in CLOSE_WAIT state for hours
Date Thu, 10 Jul 2014 19:52:11 GMT
Hey Konstantinos can you sign up for a JIRA account?
Then I can grant you permissions to add/update/edit issues.

------------------------
Chris Mattmann
chris.mattmann@gmail.com




-----Original Message-----
From: Konstantinos Mavrommatis <kmavrommatis@celgene.com>
Reply-To: <dev@oodt.apache.org>
Date: Thursday, July 10, 2014 7:11 AM
To: "dev@oodt.apache.org" <dev@oodt.apache.org>
Subject: RE: UPDATE-PROBABLY SOLVED: File manager keeps connections to
SOLR in CLOSE_WAIT state for hours

>Unless I am missing something I am not able to create an issue in JIRA :(
>Thanks
>K
>
>> -----Original Message-----
>> From: Ramirez, Paul M (398J) [mailto:paul.m.ramirez@jpl.nasa.gov]
>> Sent: Thursday, July 10, 2014 4:06 PM
>> To: <dev@oodt.apache.org>
>> Subject: Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to
>> SOLR in CLOSE_WAIT state for hours
>> 
>> Konstantinos,
>> 
>> Could you open a Jira issue for this and attach a patch? This would be
>> helpful for the project.
>> 
>> https://issues.apache.org/jira/browse/OODT/?selectedTab=com.atlassian.j
>> ira.jira-projects-plugin:summary-panel
>> 
>> 
>> Thanks,
>> Paul Ramirez
>> 
>> > On Jul 10, 2014, at 5:23 AM, "Konstantinos Mavrommatis"
>> <kmavrommatis@celgene.com> wrote:
>> >
>> > Hi,
>> >
>> > Following the suggestion at
>> >
>> > http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-
>> a
>> > void-closewait-tcp-connections/
>> >
>> >
>> >
>> > I modified the code in
>> >
>> >
>> src/main/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrClient.java
>> >
>> >
>> >
>> > and added the line highlighted in red
>> (method.setRequestHeader("Connection","close").
>> >
>> > This seems to have resolved the problem at least until now ingestion
>> of few thousand files have not produced any stale connection.
>> >
>> > Please let me know your thoughts on this solution.
>> >
>> > Best,
>> >
>> > Konstantinos
>> >
>> >
>> >
>> > ======================================================
>> >
>> > private String doHttp(HttpMethod method) throws Exception {
>> >
>> >
>> >
>> >                StringBuilder response = new StringBuilder();
>> >
>> >                BufferedReader br = null;
>> >
>> >                try {
>> >
>> >
>> >
>> >            // send request
>> >
>> >                        HttpClient httpClient = new HttpClient();
>> >
>> >                        // 10 July 2014. attempting to avoid problems
>> > with CLOSE_WAIT connections Based on
>> >
>> >
>> > //http://blogs.nuxeo.com/development/2013/02/using-httpclient-
>> properly
>> > -avoid-closewait-tcp-connections/
>> >
>> >                        method.setRequestHeader("Connection",
>> "close");
>> >
>> >            int statusCode = httpClient.executeMethod(method);
>> >
>> >
>> >
>> >            // read response
>> >
>> >            if (statusCode != HttpStatus.SC_OK) {
>> >
>> >
>> >
>> >                // still consume the response
>> >
>> >                method.getResponseBodyAsString();
>> >
>> >              throw new CatalogException("HTTP method failed: " +
>> > method.getStatusLine());
>> >
>> >
>> >
>> >            } else {
>> >
>> >
>> >
>> >                    // read the response body.
>> >
>> >                    br = new BufferedReader(new
>> > InputStreamReader(method.getResponseBodyAsStream()));
>> >
>> >        String readLine;
>> >
>> >        while(((readLine = br.readLine()) != null)) {
>> >
>> >          response.append(readLine);
>> >
>> >        }
>> >
>> >
>> >
>> > =======================================================
>> >
>> >
>> >
>> > Information about the system:
>> >
>> > OS is MacOS Mavericks,
>> >
>> > SOLR version : 4.6.1 , SOLR runs on a single server, no replication
>> at all.
>> >
>> > Tomcat verision: 7.0.50
>> >
>> > OODT version is 0.6
>> >
>> > Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >> -----Original Message-----
>> >
>> >> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
>> >
>> >> Sent: Wednesday, July 09, 2014 4:57 AM
>> >
>> >> To: dev@oodt.apache.org
>> >
>> >> Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT
>> >> state
>> >
>> >> for hours
>> >
>> >
>> >> It's just occoured to me that everything in
>> >
>> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog (well,
>> >
>> >> specifically
>> >
>> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient) is done through
>> >
>> >> HTTPClient so all of the above may not be relevant.
>> >
>> >> Can you please scope all the same?
>> >
>> >> Thanks
>> >
>> >> Lewis
>> >
>> >
>> >
>> >> On Tue, Jul 8, 2014 at 10:54 PM, Lewis John Mcgibbney <
>> >
>> >> lewis.mcgibbney@gmail.com<mailto:lewis.mcgibbney@gmail.com>> wrote:
>> >
>> >
>> >>> Hi Konstantinos,
>> >
>> >>> OK, I was ablel to scope this one and I have a few questions for
>> you.
>> >
>> >>> 1) Which version of Solr are you using? Is it  3.5, 3.6, 4.0-ALPHA?
>> >
>> >> If
>> >
>> >>> so please scope this issue [0], the solution would be to upgrade if
>> >
>> >>> you are not too long ahead with ingestion as fixes in Solr are
>> worth
>> >
>> >>> having based on recent release cycles.
>> >
>> >>> 2) How many cores do you have on Solr server? Also what kind of
>> >>> setUp
>> >
>> >>> do you have? Replication at all?
>> >
>> >>> In recent versions of Solr 4.X SolrJ clients should now call
>> >
>> >>> shutdown() on their SolrServer object to let it know they don't
>> want
>> >
>> >>> to re-use any existing connections anymore, and when Solr
>> internally
>> >
>> >>> uses SolrJ to talk to other nodes in SolrCloud it should be doing
>> >
>> >> this
>> >
>> >>> (as of 4.0-ALPHA)so this is why I ask.
>> >
>> >>> Lewis
>> >
>> >
>> >>> [0] https://issues.apache.org/jira/browse/SOLR-3280
>> >
>> >
>> >
>> >>> On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis <
>> >
>> >>> kmavrommatis@celgene.com<mailto:kmavrommatis@celgene.com>>
wrote:
>> >
>> >
>> >>>> Hi,
>> >
>> >>>> I have setup OODT filemanager on port 9000, using SOLR as the
>> >
>> >>>> indexing service on port 8081. They are both setup on the same
>> >
>> >>>> computer, while crawler runs on a number of different compute
>> nodes
>> >
>> >>>> spread across the local network and the cloud.
>> >
>> >
>> >>>> When the crawler runs and ingests files I notice that there are
>> >
>> >>>> several connections that open to solr and remain in CLOSE_WAIT
>> >>>> state
>> >
>> >> for hours.
>> >
>> >>>> any idea why this happens? Moving forward I am planning to use
>> >
>> >>>> several hundreds of crawler instances, each running on different
>> >
>> >>>> computer, that will create thousands of such connections and will
>> >
>> >>>> probably create problems to the system.
>> >
>> >>>> Thanks in advance for any help
>> >
>> >>>> Kostas
>> >
>> >
>> >>>> $lsof -i :8081
>> >
>> >>>> COMMAND   PID         USER   FD   TYPE             DEVICE SIZE/OFF
>> >
>> >> NODE
>> >
>> >>>> NAME
>> >
>> >>>> java    92065 kmavrommatis   75u  IPv6 0x392c3fa3b63b29cf      0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java    92065 kmavrommatis   76u  IPv6 0x392c3fa3b6dcbbcf      0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java    92065 kmavrommatis   77u  IPv6 0x392c3fa39fd12e0f      0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java    92065 kmavrommatis   78u  IPv6 0x392c3fa39fdcdbcf      0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java    92065 kmavrommatis   79u  IPv6 0x392c3fa3b62cde0f      0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java    92065 kmavrommatis   80u  IPv6 0x392c3fa39fa2714f      0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java    92065 kmavrommatis   81u  IPv6 0x392c3fa3b6c32acf      0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java    92065 kmavrommatis   82u  IPv6 0x392c3fa3b6aa714f      0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >
>> >
>> >>>> process 92065 is:
>> >
>> >>>> /usr/bin/java -Djava.ext.dirs=../lib
>> >
>> >>>> -Djava.util.logging.config.file=../etc/logging.properties
>> >
>> >>>> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
>> >
>> >>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum
>> 9000
>> >
>> >
>> >>>> *********************************************************
>> >
>> >>>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL
>> AND
>> >
>> >>>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE
>> >>>> USE
>> >
>> >>>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
>> >
>> >>>> If the reader is not the intended recipient, or the employee or
>> >
>> >> agent
>> >
>> >>>> responsible to deliver it to the intended recipient, you are
>> hereby
>> >
>> >>>> notified that any dissemination, distribution or copying of this
>> >
>> >>>> communication is strictly prohibited. If you have received this
>> >
>> >>>> communication in error, please reply to the sender to notify us
of
>> >
>> >>>> the error and delete the original message. Thank You.
>> >
>> >
>> >
>> >
>> >
>> >>> --
>> >
>> >>> *Lewis*
>> >
>> >
>> >
>> >
>> >
>> >> --
>> >
>> >> *Lewis*
>> >
>> > *********************************************************
>> > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
>> > MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
>> > OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
>> > If the reader is not the intended recipient, or the employee or agent
>> > responsible to deliver it to the intended recipient, you are hereby
>> > notified that any dissemination, distribution or copying of this
>> > communication is strictly prohibited. If you have received this
>> > communication in error, please reply to the sender to notify us of
>> the
>> > error and delete the original message. Thank You.
>
>*********************************************************
>THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
>CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
>INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
>OR INDIVIDUALS NAMED ABOVE.
>If the reader is not the intended recipient, or the
>employee or agent responsible to deliver it to the
>intended recipient, you are hereby notified that any
>dissemination, distribution or copying of this
>communication is strictly prohibited. If you have
>received this communication in error, please reply to the
>sender to notify us of the error and delete the original
>message. Thank You.



Mime
View raw message