Mailing-List: contact dev-help@oodt.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@oodt.apache.org
Received-SPF: pass (nike.apache.org: domain of lewis.mcgibbney@gmail.com
 designates 209.85.216.182 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <4EB788D2-2777-4043-BEE7-621249AE046C@jpl.nasa.gov>
References: 
 <0ED7EB96706C8E44B4B5B13C070262821896EA67@USSUMSPEXCMBX03.celgene.com>
	<CAGaRif3T8bVoa+vvjmv_Npjt7E5-y+QkT_T1jYWWfzr8=8Ex6w@mail.gmail.com>
	<CAGaRif0p9cXPcU31fUfi4270+-J2A1SaNL1kww094LW-o=ofjw@mail.gmail.com>
	<0ED7EB96706C8E44B4B5B13C070262821896F522@USSUMSPEXCMBX03.celgene.com>
	<62E143C4-C7FA-42CD-BCC6-2DB8C8EDAE65@jpl.nasa.gov>
	<0ED7EB96706C8E44B4B5B13C070262821896F662@USSUMSPEXCMBX03.celgene.com>
	<4EB788D2-2777-4043-BEE7-621249AE046C@jpl.nasa.gov>
Date: Thu, 10 Jul 2014 14:38:59 -0400
Message-ID: 
 <CAGaRif0pGhBeu2YS0AVH-SzLH3yBvGbfkuh6VEoyA4wb0DZ2wA@mail.gmail.com>
Subject: Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to SOLR in
 CLOSE_WAIT state for hours
From: Lewis John Mcgibbney <lewis.mcgibbney@gmail.com>
To: "dev@oodt.apache.org" <dev@oodt.apache.org>
Content-Type: multipart/alternative; boundary=001a1139a62458426004fddb2382

--001a1139a62458426004fddb2382
Content-Type: text/plain; charset=UTF-8

https://issues.apache.org/jira/browse/OODT-719
I'll ad Kos ASAP.
Thanks
Lewis


On Thu, Jul 10, 2014 at 10:50 AM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:

> Can someone log into jira and grant kostas perms?
>
> Sent from my iPhone
>
> > On Jul 10, 2014, at 7:12 AM, "Konstantinos Mavrommatis" <
> kmavrommatis@celgene.com> wrote:
> >
> > Unless I am missing something I am not able to create an issue in JIRA :(
> > Thanks
> > K
> >
> >> -----Original Message-----
> >> From: Ramirez, Paul M (398J) [mailto:paul.m.ramirez@jpl.nasa.gov]
> >> Sent: Thursday, July 10, 2014 4:06 PM
> >> To: <dev@oodt.apache.org>
> >> Subject: Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to
> >> SOLR in CLOSE_WAIT state for hours
> >>
> >> Konstantinos,
> >>
> >> Could you open a Jira issue for this and attach a patch? This would be
> >> helpful for the project.
> >>
> >> https://issues.apache.org/jira/browse/OODT/?selectedTab=com.atlassian.j
> >> ira.jira-projects-plugin:summary-panel
> >>
> >>
> >> Thanks,
> >> Paul Ramirez
> >>
> >>>> On Jul 10, 2014, at 5:23 AM, "Konstantinos Mavrommatis"
> >>> <kmavrommatis@celgene.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Following the suggestion at
> >>>
> >>> http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-
> >> a
> >>> void-closewait-tcp-connections/
> >>>
> >>>
> >>>
> >>> I modified the code in
> >> src/main/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrClient.java
> >>>
> >>>
> >>>
> >>> and added the line highlighted in red
> >> (method.setRequestHeader("Connection","close").
> >>>
> >>> This seems to have resolved the problem at least until now ingestion
> >> of few thousand files have not produced any stale connection.
> >>>
> >>> Please let me know your thoughts on this solution.
> >>>
> >>> Best,
> >>>
> >>> Konstantinos
> >>>
> >>>
> >>>
> >>> ======================================================
> >>>
> >>> private String doHttp(HttpMethod method) throws Exception {
> >>>
> >>>
> >>>
> >>>               StringBuilder response = new StringBuilder();
> >>>
> >>>               BufferedReader br = null;
> >>>
> >>>               try {
> >>>
> >>>
> >>>
> >>>           // send request
> >>>
> >>>                       HttpClient httpClient = new HttpClient();
> >>>
> >>>                       // 10 July 2014. attempting to avoid problems
> >>> with CLOSE_WAIT connections Based on
> >>>
> >>>
> >>> //http://blogs.nuxeo.com/development/2013/02/using-httpclient-
> >> properly
> >>> -avoid-closewait-tcp-connections/
> >>>
> >>>                       method.setRequestHeader("Connection",
> >> "close");
> >>>
> >>>           int statusCode = httpClient.executeMethod(method);
> >>>
> >>>
> >>>
> >>>           // read response
> >>>
> >>>           if (statusCode != HttpStatus.SC_OK) {
> >>>
> >>>
> >>>
> >>>               // still consume the response
> >>>
> >>>               method.getResponseBodyAsString();
> >>>
> >>>             throw new CatalogException("HTTP method failed: " +
> >>> method.getStatusLine());
> >>>
> >>>
> >>>
> >>>           } else {
> >>>
> >>>
> >>>
> >>>                   // read the response body.
> >>>
> >>>                   br = new BufferedReader(new
> >>> InputStreamReader(method.getResponseBodyAsStream()));
> >>>
> >>>       String readLine;
> >>>
> >>>       while(((readLine = br.readLine()) != null)) {
> >>>
> >>>         response.append(readLine);
> >>>
> >>>       }
> >>>
> >>>
> >>>
> >>> =======================================================
> >>>
> >>>
> >>>
> >>> Information about the system:
> >>>
> >>> OS is MacOS Mavericks,
> >>>
> >>> SOLR version : 4.6.1 , SOLR runs on a single server, no replication
> >> at all.
> >>>
> >>> Tomcat verision: 7.0.50
> >>>
> >>> OODT version is 0.6
> >>>
> >>> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>
> >>>> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> >>>
> >>>> Sent: Wednesday, July 09, 2014 4:57 AM
> >>>
> >>>> To: dev@oodt.apache.org
> >>>
> >>>> Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT
> >>>> state
> >>>
> >>>> for hours
> >>>
> >>>
> >>>> It's just occoured to me that everything in
> >>>
> >>>> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog (well,
> >>>
> >>>> specifically
> >>>
> >>>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient) is done through
> >>>
> >>>> HTTPClient so all of the above may not be relevant.
> >>>
> >>>> Can you please scope all the same?
> >>>
> >>>> Thanks
> >>>
> >>>> Lewis
> >>>
> >>>
> >>>
> >>>> On Tue, Jul 8, 2014 at 10:54 PM, Lewis John Mcgibbney <
> >>>
> >>>> lewis.mcgibbney@gmail.com<mailto:lewis.mcgibbney@gmail.com>> wrote:
> >>>
> >>>
> >>>>> Hi Konstantinos,
> >>>
> >>>>> OK, I was ablel to scope this one and I have a few questions for
> >> you.
> >>>
> >>>>> 1) Which version of Solr are you using? Is it  3.5, 3.6, 4.0-ALPHA?
> >>>
> >>>> If
> >>>
> >>>>> so please scope this issue [0], the solution would be to upgrade if
> >>>
> >>>>> you are not too long ahead with ingestion as fixes in Solr are
> >> worth
> >>>
> >>>>> having based on recent release cycles.
> >>>
> >>>>> 2) How many cores do you have on Solr server? Also what kind of
> >>>>> setUp
> >>>
> >>>>> do you have? Replication at all?
> >>>
> >>>>> In recent versions of Solr 4.X SolrJ clients should now call
> >>>
> >>>>> shutdown() on their SolrServer object to let it know they don't
> >> want
> >>>
> >>>>> to re-use any existing connections anymore, and when Solr
> >> internally
> >>>
> >>>>> uses SolrJ to talk to other nodes in SolrCloud it should be doing
> >>>
> >>>> this
> >>>
> >>>>> (as of 4.0-ALPHA)so this is why I ask.
> >>>
> >>>>> Lewis
> >>>
> >>>
> >>>>> [0] https://issues.apache.org/jira/browse/SOLR-3280
> >>>
> >>>
> >>>
> >>>>> On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis <
> >>>
> >>>>> kmavrommatis@celgene.com<mailto:kmavrommatis@celgene.com>> wrote:
> >>>
> >>>
> >>>>>> Hi,
> >>>
> >>>>>> I have setup OODT filemanager on port 9000, using SOLR as the
> >>>
> >>>>>> indexing service on port 8081. They are both setup on the same
> >>>
> >>>>>> computer, while crawler runs on a number of different compute
> >> nodes
> >>>
> >>>>>> spread across the local network and the cloud.
> >>>
> >>>
> >>>>>> When the crawler runs and ingests files I notice that there are
> >>>
> >>>>>> several connections that open to solr and remain in CLOSE_WAIT
> >>>>>> state
> >>>
> >>>> for hours.
> >>>
> >>>>>> any idea why this happens? Moving forward I am planning to use
> >>>
> >>>>>> several hundreds of crawler instances, each running on different
> >>>
> >>>>>> computer, that will create thousands of such connections and will
> >>>
> >>>>>> probably create problems to the system.
> >>>
> >>>>>> Thanks in advance for any help
> >>>
> >>>>>> Kostas
> >>>
> >>>
> >>>>>> $lsof -i :8081
> >>>
> >>>>>> COMMAND   PID         USER   FD   TYPE             DEVICE SIZE/OFF
> >>>
> >>>> NODE
> >>>
> >>>>>> NAME
> >>>
> >>>>>> java    92065 kmavrommatis   75u  IPv6 0x392c3fa3b63b29cf      0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java    92065 kmavrommatis   76u  IPv6 0x392c3fa3b6dcbbcf      0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java    92065 kmavrommatis   77u  IPv6 0x392c3fa39fd12e0f      0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java    92065 kmavrommatis   78u  IPv6 0x392c3fa39fdcdbcf      0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java    92065 kmavrommatis   79u  IPv6 0x392c3fa3b62cde0f      0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java    92065 kmavrommatis   80u  IPv6 0x392c3fa39fa2714f      0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java    92065 kmavrommatis   81u  IPv6 0x392c3fa3b6c32acf      0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java    92065 kmavrommatis   82u  IPv6 0x392c3fa3b6aa714f      0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>
> >>>
> >>>>>> process 92065 is:
> >>>
> >>>>>> /usr/bin/java -Djava.ext.dirs=../lib
> >>>
> >>>>>> -Djava.util.logging.config.file=../etc/logging.properties
> >>>
> >>>>>> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
> >>>
> >>>>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum
> >> 9000
> >>>
> >>>
> >>>>>> *********************************************************
> >>>
> >>>>>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL
> >> AND
> >>>
> >>>>>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE
> >>>>>> USE
> >>>
> >>>>>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> >>>
> >>>>>> If the reader is not the intended recipient, or the employee or
> >>>
> >>>> agent
> >>>
> >>>>>> responsible to deliver it to the intended recipient, you are
> >> hereby
> >>>
> >>>>>> notified that any dissemination, distribution or copying of this
> >>>
> >>>>>> communication is strictly prohibited. If you have received this
> >>>
> >>>>>> communication in error, please reply to the sender to notify us of
> >>>
> >>>>>> the error and delete the original message. Thank You.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>> --
> >>>
> >>>>> *Lewis*
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> --
> >>>
> >>>> *Lewis*
> >>>
> >>> *********************************************************
> >>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
> >>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
> >>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> >>> If the reader is not the intended recipient, or the employee or agent
> >>> responsible to deliver it to the intended recipient, you are hereby
> >>> notified that any dissemination, distribution or copying of this
> >>> communication is strictly prohibited. If you have received this
> >>> communication in error, please reply to the sender to notify us of
> >> the
> >>> error and delete the original message. Thank You.
> >
> > *********************************************************
> > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
> > CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
> > INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
> > OR INDIVIDUALS NAMED ABOVE.
> > If the reader is not the intended recipient, or the
> > employee or agent responsible to deliver it to the
> > intended recipient, you are hereby notified that any
> > dissemination, distribution or copying of this
> > communication is strictly prohibited. If you have
> > received this communication in error, please reply to the
> > sender to notify us of the error and delete the original
> > message. Thank You.
>


-- 
*Lewis*

--001a1139a62458426004fddb2382--