Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 86048 invoked from network); 13 Nov 2009 10:34:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 13 Nov 2009 10:34:22 -0000 Received: (qmail 87387 invoked by uid 500); 13 Nov 2009 10:34:20 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 87309 invoked by uid 500); 13 Nov 2009 10:34:19 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 87299 invoked by uid 99); 13 Nov 2009 10:34:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Nov 2009 10:34:19 +0000 X-ASF-Spam-Status: No, hits=-2.6 required=5.0 tests=AWL,BAYES_00 X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [209.85.211.183] (HELO mail-yw0-f183.google.com) (209.85.211.183) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Nov 2009 10:34:16 +0000 Received: by ywh13 with SMTP id 13so3547566ywh.29 for ; Fri, 13 Nov 2009 02:33:54 -0800 (PST) MIME-Version: 1.0 Received: by 10.150.250.13 with SMTP id x13mr7463226ybh.230.1258108433828; Fri, 13 Nov 2009 02:33:53 -0800 (PST) In-Reply-To: <26334253.post@talk.nabble.com> References: <8837fb770911081941v5e94066eg275c55ca1e46e3fb@mail.gmail.com> <9ac0c6aa0911090243n785fc8faj1c76665721299bb@mail.gmail.com> <26298910.post@talk.nabble.com> <9ac0c6aa0911110215t3462ad07s7d0ee455ad21a958@mail.gmail.com> <26299405.post@talk.nabble.com> <9ac0c6aa0911110259y411f295fx431f210e462cd8d8@mail.gmail.com> <26299941.post@talk.nabble.com> <85d3c3b60911121015h52a69840sead79f1b05fc2613@mail.gmail.com> <26334253.post@talk.nabble.com> Date: Fri, 13 Nov 2009 05:33:53 -0500 Message-ID: <9ac0c6aa0911130233xb173d36xf2b3d39eb2b0a654@mail.gmail.com> Subject: Re: IndexWriter.close() no longer seems to close everything From: Michael McCandless To: java-user@lucene.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Any luck narrowing this to a standalone test case that shows the problem? That new exception appears to be inside the Java code created by the app server compiling your JSP -- it's not very helpful since it doesn't "enter" Lucene. Can you try to narrow this to a standalone test case, too? Thanks. Mike On Fri, Nov 13, 2009 at 5:12 AM, Albert Juhe wrote: > > Hi, > > About this problem I did a test yesterday, I did a downgrade, I changed > versi=F3n 2.9.1 to 2.4.1, and the problem has been solved, all the files = are > closed corretly and JBOSS isn't unstable. > > Another problem that we have observed is: > > Sometimes, random success, when you try to make a serach the first time > don't return result but the second time yes (Lucene 2.9.1. BooleanQuery). > I get this error message: > java.lang.NullPointerException > =A0 =A0 =A0 =A0at > org.apache.jsp.objects.buscar_005fresposta_jsp._jspService(buscar_005fres= posta_jsp.java:631) > =A0 =A0 =A0 =A0at > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) > =A0 =A0 =A0 =A0at javax.servlet.http.HttpServlet.service(HttpServlet.java= :810) > =A0 =A0 =A0 =A0at > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.jav= a:334) > =A0 =A0 =A0 =A0at > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) > =A0 =A0 =A0 =A0at org.apache.jasper.servlet.JspServlet.service(JspServlet= .java:264) > =A0 =A0 =A0 =A0at javax.servlet.http.HttpServlet.service(HttpServlet.java= :810) > =A0 =A0 =A0 =A0at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicat= ionFilterChain.java:252) > =A0 =A0 =A0 =A0at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilte= rChain.java:173) > =A0 =A0 =A0 =A0at > org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter= .java:96) > =A0 =A0 =A0 =A0at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicat= ionFilterChain.java:202) > =A0 =A0 =A0 =A0at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilte= rChain.java:173) > =A0 =A0 =A0 =A0at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve= .java:213) > =A0 =A0 =A0 =A0at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve= .java:178) > =A0 =A0 =A0 =A0at > org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAss= ociationValve.java:175) > =A0 =A0 =A0 =A0at > org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.ja= va:74) > =A0 =A0 =A0 =A0at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:= 126) > =A0 =A0 =A0 =A0at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:= 105) > =A0 =A0 =A0 =A0at > org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectio= nValve.java:156) > =A0 =A0 =A0 =A0at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.j= ava:107) > =A0 =A0 =A0 =A0at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:14= 8) > =A0 =A0 =A0 =A0at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869= ) > =A0 =A0 =A0 =A0at > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.proce= ssConnection(Http11BaseProtocol.java:664) > =A0 =A0 =A0 =A0at > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.= java:527) > =A0 =A0 =A0 =A0at > org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerT= hread.java:112) > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:595) > > I don't know whats the meanig of it, but buscar_resposta.jsp is the jsp t= hat > makes a Search. > > With 2.4.1 this problem don't happens. > > > Jason Rutherglen-2 wrote: >> >> If there's a bug you're seeing, it's helpful to open an issue and post >> code reproducing it. >> >> On Wed, Nov 11, 2009 at 3:41 AM, Albert Juhe wrot= e: >>> >>> I think that this is the best way to proceed. >>> >>> thank you Mike >>> >>> >>> >>> Michael McCandless-2 wrote: >>>> >>>> Can you narrow the leak down to a small self-contained test? >>>> >>>> Mike >>>> >>>> On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe >>>> wrote: >>>>> >>>>> I don't get any exception. >>>>> >>>>> thank you Mike >>>>> >>>>> >>>>> Michael McCandless-2 wrote: >>>>>> >>>>>> Do you see your exception handler printing anything out? >>>>>> >>>>>> You don't need to close the underlying IndexReader, just the >>>>>> IndexSearcher (which will close the IndexReader, since it was the on= e >>>>>> that had opened it). >>>>>> >>>>>> Mike >>>>>> >>>>>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe >>>>>> wrote: >>>>>>> >>>>>>> I don't know if it's the same problem but I think it's similar, >>>>>>> >>>>>>> My problem is with the Indexsearcher. I've installed a web search >>>>>>> engine >>>>>>> that uses Lucene, after a search I make a close operation like this= : >>>>>>> >>>>>>> private IndexSearcher searcher; >>>>>>> >>>>>>> NIOFSDirectory directory =3D new NIOFSDirectory(new File(path)); >>>>>>> this.searcher =3D new IndexSearcher(directory, true); >>>>>>> >>>>>>> =A0public void close() { >>>>>>> =A0 =A0 =A0 =A0try { >>>>>>> =A0 =A0 =A0 =A0 =A0 =A0System.out.println("Closing: " >>>>>>> +this.searcher.getIndexReader().directory().toString() ); >>>>>>> =A0 =A0 =A0 =A0 =A0 =A0this.searcher.getIndexReader().directory().c= lose(); >>>>>>> =A0 =A0 =A0 =A0 =A0 =A0this.searcher.getIndexReader().close(); >>>>>>> =A0 =A0 =A0 =A0 =A0 =A0this.searcher.close(); >>>>>>> =A0 =A0 =A0 =A0} catch (IOException e) { >>>>>>> =A0 =A0 =A0 =A0 =A0 =A0System.out.println(" caught a " + e.getClass= () + "\n with >>>>>>> message: " + e.getMessage()); >>>>>>> =A0 =A0 =A0 =A0} >>>>>>> =A0 =A0} >>>>>>> jboss don't close *.cfs index files, everytime that I make a new >>>>>>> search >>>>>>> a >>>>>>> new file is caught by jboss. >>>>>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof comman= d, >>>>>>> the >>>>>>> only way to free this files and close it is stopping jboss. >>>>>>> >>>>>>> The problem is that after search many times, =A0jboss go down and I >>>>>>> have >>>>>>> to >>>>>>> restart it. >>>>>>> >>>>>>> Do you have the same problem with IndexSearch or is only with >>>>>>> IndexWriter? >>>>>>> >>>>>>> Albert Juhe >>>>>>> Learning Technologies >>>>>>> Universitat Oberta de Catalunya >>>>>>> >>>>>>> >>>>>>> Michael McCandless-2 wrote: >>>>>>>> >>>>>>>> Does this look like a real leak John? =A0You're definitely closing >>>>>>>> every >>>>>>>> reader you get back from getReader? >>>>>>>> >>>>>>>> Mike >>>>>>>> >>>>>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang >>>>>>>> wrote: >>>>>>>>> I am seeing the samething, but only when IndexWriter.getReader is >>>>>>>>> called >>>>>>>>> at >>>>>>>>> a high rate. >>>>>>>>> >>>>>>>>> from lsof, I see file handles growing. >>>>>>>>> >>>>>>>>> -John >>>>>>>>> >>>>>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi all. >>>>>>>>>> >>>>>>>>>> We updated to Lucene 2.9, and now we find that after closing our >>>>>>>>>> text >>>>>>>>>> index, it is not possible to rename the directory in which it >>>>>>>>>> resides >>>>>>>>>> (we are actually renaming a directory further up the hierarchy.) >>>>>>>>>> >>>>>>>>>> We discovered that the following files were still open by the >>>>>>>>>> process: >>>>>>>>>> >>>>>>>>>> =A0_0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tv= f >>>>>>>>>> >>>>>>>>>> We are calling IndexWriter.close() shortly before attempting to >>>>>>>>>> write >>>>>>>>>> to the directory (a few lines of code earlier) so I suspect it >>>>>>>>>> could >>>>>>>>>> be related to timing somehow if Lucene is perhaps still doing >>>>>>>>>> something on a background thread at this time (though I was unde= r >>>>>>>>>> the >>>>>>>>>> impression that close() waited for merges and so forth to comple= te >>>>>>>>>> before returning.) >>>>>>>>>> >>>>>>>>>> Daniel >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Daniel Noll =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0Forensic and eDiscovery >>>>>>>>>> Software >>>>>>>>>> Senior Developer =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0The world's most >>>>>>>>>> advanced >>>>>>>>>> Nuix =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0email data >>>>>>>>>> analysis >>>>>>>>>> http://nuix.com/ =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0and eDiscovery >>>>>>>>>> software >>>>>>>>>> >>>>>>>>>> ----------------------------------------------------------------= ----- >>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.or= g >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> ------------------------------------------------------------------= --- >>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> View this message in context: >>>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-cl= ose-everything-tp26260801p26298910.html >>>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.co= m. >>>>>>> >>>>>>> >>>>>>> -------------------------------------------------------------------= -- >>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org >>>>>>> >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------= - >>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-clos= e-everything-tp26260801p26299405.html >>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com. >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>>>> For additional commands, e-mail: java-user-help@lucene.apache.org >>>>> >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>>> For additional commands, e-mail: java-user-help@lucene.apache.org >>>> >>>> >>>> >>> >>> -- >>> View this message in context: >>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-= everything-tp26260801p26299941.html >>> Sent from the Lucene - Java Users mailing list archive at Nabble.com. >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >>> For additional commands, e-mail: java-user-help@lucene.apache.org >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org >> For additional commands, e-mail: java-user-help@lucene.apache.org >> >> >> > > -- > View this message in context: http://old.nabble.com/IndexWriter.close%28%= 29-no-longer-seems-to-close-everything-tp26260801p26334253.html > Sent from the Lucene - Java Users mailing list archive at Nabble.com. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-user-help@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org