lucenenet-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "George Aroush" <geo...@aroush.net>
Subject RE: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
Date Mon, 25 Sep 2006 02:19:02 GMT
Hi Michael,

That is right.  If you take the current code as-is (without the change to
Close() as I mentioned earlier) of Lucene.Net 1.9, 1.9.1 and 2.0 and compile
them using .NET 2.0, you will not see any leak!

What's even more puzzling to me is this.  If I add the following code: 

	~RAMOutputStream()
	{
		file = null;
	}
 
To the class Lucene.Net.Store.RAMOutputStream, the leak more then doubles!!

-- George Aroush

-----Original Message-----
From: Michael Mitiaguin [mailto:mitiaguin@gmail.com]
Sent: Sunday, September 24, 2006 9:59 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: Memory leak (was: RE: Lucene.Net Indexing Large Databases)

George,

Sorry for naive question how does it relate to your comment

"If I compile Lucene.Net using VC8 (i.e.: .NET 2.0) the leak won't occur.
In addition, with .NET 2.0, the destructor/finalize of TermInfosReader is
getting called, but not when compiled with .NET 1.1"

Close() as is just doesn't cause any leak in Net 2.0 ?

Regards
Michael Mitiaguin

On 9/25/06, George Aroush <george@aroush.net> wrote:
>
> Hi Narendra and all,
>
> Here is what I have so far.
>
> The leak is coming from two classes: Lucene.Net.Store.RAMInputStream 
> and Lucene.Net.Store.RAMOutputStream.  Both of those classes are 
> leaking because the instances of Lucene.Net.Store.RAMFile (which both
> create) are not being released.
>
> In the class Lucene.Net.Store.RAMInputStream, if I modify the function
> Close() from:
>
>         public override void  Close()
>         {
>         }
>
> To:
>
>         public override void  Close()
>         {
>                 file.buffers.Clear();   // I added this line
>         }
>
> It will cut down the memory leak by up to 40%.
>
> A similar code change in RAMOutputStream will result with a crash.
>
> It's clear to me that instances of RAMInputStream and RAMOutputStream 
> are not being released (because of RAMFile) thus, the leak.  I don't 
> believe the line that I added is the fix.
>
> So, I am still digging but any help anyone can offer would be great.
>
> Regards,
>
> -- George Aroush
>
> -----Original Message-----
> From: Narendra Tumkur [mailto:ntumkur@drive.com.au]
> Sent: Sunday, September 24, 2006 7:05 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: Memory leak (was: RE: Lucene.Net Indexing Large
> Databases)
>
> Hi George,
> Any news on this memory leak ?
>
> Narendra Tumkur
> Web Developer
>
> drive.com.au
>
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Wednesday, 13 September 2006 8:01 AM
> To: lucene-net-dev@incubator.apache.org;
> lucene-net-user@incubator.apache.org
> Subject: RE: Memory leak (was: RE: Lucene.Net Indexing Large
> Databases)
>
> Hi Folks,
>
> Some more update on the memory leak.
>
> If I compile Lucene.Net using VC8 (i.e.: .NET 2.0) the leak won't 
> occur.  In addition, with .NET 2.0, the destructor/finalize of 
> TermInfosReader is getting called, but not when compiled with .NET 
> 1.1.
>
> I checked the Java version, and there too, TermInfosReader.finalize() 
> is getting called.
>
> No, the answer is not to simply switch over to .NET 2.0, I have a need 
> to run Lucene.Net on .NET 1.1 and I am sure many others do too.
>
> Regards,
>
> -- George
>
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Monday, September 11, 2006 9:34 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
>
> No, adding directory.Close() will not do the trick.  However, I do 
> believe we are missing a Close() somewhere, most likely to 
> SegmentReader or TermInfosReader.
>
> All indication are that one or both of those classes is the source of 
> the leak.  Those were my suspects and the MS CLR Profiler is pointing 
> in this direction.
>
> The Java version lands you into the Finalize of those two classes, but 
> with the C# version, the Finalize of those two classes is never 
> getting called!
>
> I hope to find out more in the next few days, but if anyone got the 
> cycles and got a nice tool to help with this defect, please give this 
> a try and let me know what you find.
>
> Regards,
>
> -- George
>
> -----Original Message-----
> From: Eyal Post [mailto:eyalpost@epocalipse.com]
> Sent: Monday, September 11, 2006 6:50 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Adding:
> directory.Close();
> At the end of the method should help. Do you see still see leaks after 
> that?
>
> Eyal
>
>
> > -----Original Message-----
> > From: George Aroush [mailto:george@aroush.net]
> > Sent: Monday, September 11, 2006 3:50 AM
> > To: lucene-net-user@incubator.apache.org;
> > lucene-net-dev@incubator.apache.org
> > Subject: RE: Lucene.Net Indexing Large Databases
> >
> > Hi Folks,
> >
> > Since last weekend, I have been trying to narrow down the problem to 
> > this memory leak without much of a luck.
> >
> > Does anyone have a tool (or could recommend one, without costing me
> > $$) which hopefully show the source of the leak?
> >
> > Unlike C++ code, the leak here, obviously, is due to not releasing 
> > references to temporary or real objects.  The trick is finding the 
> > object.
> >
> > This leak can be created with this simple code:
> >
> >       public static void  Main(System.String[] args)
> >       {
> >               IndexWriter diskIndex;
> >               Directory   directory;
> >               Analyzer    analyzer;
> >               Document    doc;
> >               int         count;
> >               string      indexDirectory;
> >               System.IO.FileInfo  fi;
> >
> >               indexDirectory = "C:\\Index.Bad";
> >
> >               fi = new System.IO.FileInfo(indexDirectory);
> >               directory =
> > Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> > true);
> >
> >               analyzer = new SimpleAnalyzer();
> >               diskIndex = new IndexWriter(directory, analyzer, 
> > true);
> >
> >               count = 0;
> >               while (count < 10000)
> >               {
> >                       doc = new Document();
> >                       diskIndex.AddDocument(doc);
> >                       count++;
> >               }
> >
> >               diskIndex.Close();
> >       }
> >
> > This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3.  I 
> > also verified and it doesn't leak under the Java version of Lucene 
> > (2.0 is where I tested.)
> >
> > Regards,
> >
> > -- George
> >
> >
> > -----Original Message-----
> > From: George Aroush [mailto:george@aroush.net]
> > Sent: Friday, September 01, 2006 9:21 PM
> > To: lucene-net-user@incubator.apache.org
> > Subject: RE: Lucene.Net Indexing Large Databases
> >
> > Hi Chris,
> >
> > I am using 1.9.1 in production and I am not having this problem.
> > Sorry, I don't have enough cycles to try your code on 1.9.
> >
> > This problem was reported on 1.4.x and was fixed.  I am sure I 
> > carried it over to 1.9.x and 2.0 -- or maybe this is a new issue.  I 
> > will double check when I get the cycles.
> >
> > You can get 1.4.3's source code as ZIP from the download site of 
> > Lucene.Net which is here:
> > https://svn.apache.org/repos/asf/incubator/lucene.net/site/dow
> nload/ or you can SVN the source code from here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
>
> Regards,
>
> -- George Aroush
>
>
> -----Original Message-----
> From: Chris David [mailto:chris@snapstream.com]
> Sent: Friday, September 01, 2006 1:46 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Thanks René, so its not just me with this problem.  Now where can I 
> get a hold of this wonderful 1.4 Build of Lucene.  Its not listed 
> directly on Apache's Lucene.NET Page. I am anxious to see if my code 
> actually does work.
> Thanks again for all your help, I really do appreciate it.
>
> Chris
> Snapstream Media
> -----Original Message-----
> From: René de Vries [mailto:rene.de.vries@howardshome.com]
> Sent: Friday, September 01, 2006 7:32 AM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Update: I didn't realize my earlier code example ran against 1.4.
>
> If I run this with 1.9final-005 build, I am experiencing the exact 
> same problems as Chris mentions. Memory consumtion keeps growing, I 
> had to kill it at 1.5Gb. Exact same code, but with a 1.4 version of 
> the lucene.netDLL, and it runs along at 50Mb
>
> René
>
>
>
>


Mime
View raw message