Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 72992 invoked from network); 13 Nov 2009 10:13:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 13 Nov 2009 10:13:35 -0000 Received: (qmail 54480 invoked by uid 500); 13 Nov 2009 10:13:33 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 54410 invoked by uid 500); 13 Nov 2009 10:13:33 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 54076 invoked by uid 99); 13 Nov 2009 10:13:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Nov 2009 10:13:32 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of lists@nabble.com designates 216.139.236.158 as permitted sender) Received: from [216.139.236.158] (HELO kuber.nabble.com) (216.139.236.158) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Nov 2009 10:13:20 +0000 Received: from isper.nabble.com ([192.168.236.156]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from ) id 1N8t9H-0003dJ-VQ for java-user@lucene.apache.org; Fri, 13 Nov 2009 02:12:59 -0800 Message-ID: <26334253.post@talk.nabble.com> Date: Fri, 13 Nov 2009 02:12:59 -0800 (PST) From: Albert Juhe To: java-user@lucene.apache.org Subject: Re: IndexWriter.close() no longer seems to close everything In-Reply-To: <85d3c3b60911121015h52a69840sead79f1b05fc2613@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Nabble-From: albertjuhe@gmail.com References: <8837fb770911081941v5e94066eg275c55ca1e46e3fb@mail.gmail.com> <9ac0c6aa0911090243n785fc8faj1c76665721299bb@mail.gmail.com> <26298910.post@talk.nabble.com> <9ac0c6aa0911110215t3462ad07s7d0ee455ad21a958@mail.gmail.com> <26299405.post@talk.nabble.com> <9ac0c6aa0911110259y411f295fx431f210e462cd8d8@mail.gmail.com> <26299941.post@talk.nabble.com> <85d3c3b60911121015h52a69840sead79f1b05fc2613@mail.gmail.com> X-Virus-Checked: Checked by ClamAV on apache.org Hi, About this problem I did a test yesterday, I did a downgrade, I changed versi=C3=B3n 2.9.1 to 2.4.1, and the problem has been solved, all the files= are closed corretly and JBOSS isn't unstable. Another problem that we have observed is: Sometimes, random success, when you try to make a serach the first time don't return result but the second time yes (Lucene 2.9.1. BooleanQuery). I get this error message: java.lang.NullPointerException at org.apache.jsp.objects.buscar_005fresposta_jsp._jspService(buscar_005frespo= sta_jsp.java:631) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:= 334) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264= ) at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio= nFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC= hain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.j= ava:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio= nFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC= hain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j= ava:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j= ava:178) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssoc= iationValve.java:175) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java= :74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12= 6) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10= 5) at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionV= alve.java:156) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav= a:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.process= Connection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.ja= va:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThr= ead.java:112) at java.lang.Thread.run(Thread.java:595) I don't know whats the meanig of it, but buscar_resposta.jsp is the jsp tha= t makes a Search. With 2.4.1 this problem don't happens. Jason Rutherglen-2 wrote: >=20 > If there's a bug you're seeing, it's helpful to open an issue and post > code reproducing it. >=20 > On Wed, Nov 11, 2009 at 3:41 AM, Albert Juhe wrote= : >> >> I think that this is the best way to proceed. >> >> thank you Mike >> >> >> >> Michael McCandless-2 wrote: >>> >>> Can you narrow the leak down to a small self-contained test? >>> >>> Mike >>> >>> On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe >>> wrote: >>>> >>>> I don't get any exception. >>>> >>>> thank you Mike >>>> >>>> >>>> Michael McCandless-2 wrote: >>>>> >>>>> Do you see your exception handler printing anything out? >>>>> >>>>> You don't need to close the underlying IndexReader, just the >>>>> IndexSearcher (which will close the IndexReader, since it was the one >>>>> that had opened it). >>>>> >>>>> Mike >>>>> >>>>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe >>>>> wrote: >>>>>> >>>>>> I don't know if it's the same problem but I think it's similar, >>>>>> >>>>>> My problem is with the Indexsearcher. I've installed a web search >>>>>> engine >>>>>> that uses Lucene, after a search I make a close operation like this: >>>>>> >>>>>> private IndexSearcher searcher; >>>>>> >>>>>> NIOFSDirectory directory =3D new NIOFSDirectory(new File(path)); >>>>>> this.searcher =3D new IndexSearcher(directory, true); >>>>>> >>>>>> =C2=A0public void close() { >>>>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0try { >>>>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0System.out.println("Closing= : " >>>>>> +this.searcher.getIndexReader().directory().toString() ); >>>>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0this.searcher.getIndexReade= r().directory().close(); >>>>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0this.searcher.getIndexReade= r().close(); >>>>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0this.searcher.close(); >>>>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0} catch (IOException e) { >>>>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0System.out.println(" caught= a " + e.getClass() + "\n with >>>>>> message: " + e.getMessage()); >>>>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0} >>>>>> =C2=A0 =C2=A0} >>>>>> jboss don't close *.cfs index files, everytime that I make a new >>>>>> search >>>>>> a >>>>>> new file is caught by jboss. >>>>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command= , >>>>>> the >>>>>> only way to free this files and close it is stopping jboss. >>>>>> >>>>>> The problem is that after search many times, =C2=A0jboss go down and= I >>>>>> have >>>>>> to >>>>>> restart it. >>>>>> >>>>>> Do you have the same problem with IndexSearch or is only with >>>>>> IndexWriter? >>>>>> >>>>>> Albert Juhe >>>>>> Learning Technologies >>>>>> Universitat Oberta de Catalunya >>>>>> >>>>>> >>>>>> Michael McCandless-2 wrote: >>>>>>> >>>>>>> Does this look like a real leak John? =C2=A0You're definitely closi= ng >>>>>>> every >>>>>>> reader you get back from getReader? >>>>>>> >>>>>>> Mike >>>>>>> >>>>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang >>>>>>> wrote: >>>>>>>> I am seeing the samething, but only when IndexWriter.getReader is >>>>>>>> called >>>>>>>> at >>>>>>>> a high rate. >>>>>>>> >>>>>>>> from lsof, I see file handles growing. >>>>>>>> >>>>>>>> -John >>>>>>>> >>>>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi all. >>>>>>>>> >>>>>>>>> We updated to Lucene 2.9, and now we find that after closing our >>>>>>>>> text >>>>>>>>> index, it is not possible to rename the directory in which it >>>>>>>>> resides >>>>>>>>> (we are actually renaming a directory further up the hierarchy.) >>>>>>>>> >>>>>>>>> We discovered that the following files were still open by the >>>>>>>>> process: >>>>>>>>> >>>>>>>>> =C2=A0_0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.= tvf >>>>>>>>> >>>>>>>>> We are calling IndexWriter.close() shortly before attempting to >>>>>>>>> write >>>>>>>>> to the directory (a few lines of code earlier) so I suspect it >>>>>>>>> could >>>>>>>>> be related to timing somehow if Lucene is perhaps still doing >>>>>>>>> something on a background thread at this time (though I was under >>>>>>>>> the >>>>>>>>> impression that close() waited for merges and so forth to complet= e >>>>>>>>> before returning.) >>>>>>>>> >>>>>>>>> Daniel >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Daniel Noll =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Forensic and eDiscovery >>>>>>>>> Software >>>>>>>>> Senior Developer =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The world's most >>>>>>>>> advanced >>>>>>>>> Nuix =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0email data >>>>>>>>> analysis >>>>>>>>> http://nuix.com/ =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0and eDiscove= ry >>>>>>>>> software >>>>>>>>> >>>>>>>>> -----------------------------------------------------------------= ---- >>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -------------------------------------------------------------------= -- >>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-clo= se-everything-tp26260801p26298910.html >>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com= . >>>>>> >>>>>> >>>>>> --------------------------------------------------------------------= - >>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org >>>>>> >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>>>> For additional commands, e-mail: java-user-help@lucene.apache.org >>>>> >>>>> >>>>> >>>> >>>> -- >>>> View this message in context: >>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close= -everything-tp26260801p26299405.html >>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com. >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>>> For additional commands, e-mail: java-user-help@lucene.apache.org >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>> For additional commands, e-mail: java-user-help@lucene.apache.org >>> >>> >>> >> >> -- >> View this message in context: >> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-e= verything-tp26260801p26299941.html >> Sent from the Lucene - Java Users mailing list archive at Nabble.com. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >> For additional commands, e-mail: java-user-help@lucene.apache.org >> >> >=20 > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-user-help@lucene.apache.org >=20 >=20 >=20 --=20 View this message in context: http://old.nabble.com/IndexWriter.close%28%29= -no-longer-seems-to-close-everything-tp26260801p26334253.html Sent from the Lucene - Java Users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org