lucenenet-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Rodenburg" <jeff.rodenb...@gmail.com>
Subject Re: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
Date Tue, 12 Sep 2006 01:58:46 GMT
The Close() method addition in the 1.4.3 code was in the FieldCache class.
The difference was that the implementation in Java used a WeakHashMap vs.
the C# version used a straight Hashtable.  Java's WeakHashMap uses weak
references; the Hashtable has strong references.  GC wasn't able to clear
the FieldCache, so a virtual Close() method with calls to explicitly release
references from internal Hashtable was added.

-- j


On 9/11/06, George Aroush <george@aroush.net> wrote:
>
> Hi Eyal,
>
> If you run this sample program (see email thread) using Lucene.Net 1.4.3you
> will not see any memory growth, but if you try with 1.9, 1.9.1 and 2.0,
> memory will consistently grow.
>
> Also, if you port this program to the Java version, again, using 1.4.3 or
> 2.0 (I didn't try 1.9 or 1.9.1) you won't see any memory growth.
>
> Clearly, we have a leak somewhere in Lucene.Net 1.9, 1.9.1 and 2.0.  There
> was a similar problem with 1.4.3 and the fix was adding a Close() --
> sorry,
> I don't remember in which method, but if you are interested you can look
> it
> up on dotLucne when it was at SourceForge.  I should point out that the
> Java
> version doesn't need the Close() call.
>
> Regards,
>
> -- George
>
> -----Original Message-----
> From: Eyal Post [mailto:eyalpost@epocalipse.com]
> Sent: Monday, September 11, 2006 1:19 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi George,
>
> Can you explain how do you see that there's actually a leak?
>
> Eyal
>
> > -----Original Message-----
> > From: George Aroush [mailto:george@aroush.net]
> > Sent: Monday, September 11, 2006 3:50 AM
> > To: lucene-net-user@incubator.apache.org;
> > lucene-net-dev@incubator.apache.org
> > Subject: RE: Lucene.Net Indexing Large Databases
> >
> > Hi Folks,
> >
> > Since last weekend, I have been trying to narrow down the problem to
> > this memory leak without much of a luck.
> >
> > Does anyone have a tool (or could recommend one, without costing me
> > $$) which hopefully show the source of the leak?
> >
> > Unlike C++ code, the leak here, obviously, is due to not releasing
> > references to temporary or real objects.  The trick is finding the
> > object.
> >
> > This leak can be created with this simple code:
> >
> >       public static void  Main(System.String[] args)
> >       {
> >               IndexWriter diskIndex;
> >               Directory   directory;
> >               Analyzer    analyzer;
> >               Document    doc;
> >               int         count;
> >               string      indexDirectory;
> >               System.IO.FileInfo  fi;
> >
> >               indexDirectory = "C:\\Index.Bad";
> >
> >               fi = new System.IO.FileInfo(indexDirectory);
> >               directory =
> > Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> > true);
> >
> >               analyzer = new SimpleAnalyzer();
> >               diskIndex = new IndexWriter(directory, analyzer, true);
> >
> >               count = 0;
> >               while (count < 10000)
> >               {
> >                       doc = new Document();
> >                       diskIndex.AddDocument(doc);
> >                       count++;
> >               }
> >
> >               diskIndex.Close();
> >       }
> >
> > This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3.  I
> > also verified and it doesn't leak under the Java version of Lucene
> > (2.0 is where I tested.)
> >
> > Regards,
> >
> > -- George
> >
> >
> > -----Original Message-----
> > From: George Aroush [mailto:george@aroush.net]
> > Sent: Friday, September 01, 2006 9:21 PM
> > To: lucene-net-user@incubator.apache.org
> > Subject: RE: Lucene.Net Indexing Large Databases
> >
> > Hi Chris,
> >
> > I am using 1.9.1 in production and I am not having this problem.
> > Sorry, I don't have enough cycles to try your code on 1.9.
> >
> > This problem was reported on 1.4.x and was fixed.  I am sure I carried
> > it over to 1.9.x and 2.0 -- or maybe this is a new issue.  I will
> > double check when I get the cycles.
> >
> > You can get 1.4.3's source code as ZIP from the download site of
> > Lucene.Net which is here:
> > https://svn.apache.org/repos/asf/incubator/lucene.net/site/dow
> nload/ or you can SVN the source code from here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
>
> Regards,
>
> -- George Aroush
>
>
> -----Original Message-----
> From: Chris David [mailto:chris@snapstream.com]
> Sent: Friday, September 01, 2006 1:46 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Thanks René, so its not just me with this problem.  Now where can I get a
> hold of this wonderful 1.4 Build of Lucene.  Its not listed directly on
> Apache's Lucene.NET Page. I am anxious to see if my code actually does
> work.
> Thanks again for all your help, I really do appreciate it.
>
> Chris
> Snapstream Media
> -----Original Message-----
> From: René de Vries [mailto:rene.de.vries@howardshome.com]
> Sent: Friday, September 01, 2006 7:32 AM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Update: I didn't realize my earlier code example ran against 1.4.
>
> If I run this with 1.9final-005 build, I am experiencing the exact same
> problems as Chris mentions. Memory consumtion keeps growing, I had to kill
> it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.netDLL,
> and it runs along at 50Mb
>
> René
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message