lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From adrena.keat...@nl.abnamro.com
Subject Re: Lucene/Tomcat Memory Leak Issue
Date Wed, 16 Aug 2006 16:33:11 GMT
We are indexing the file server via Lucene.  We have a 15 MB index file and
have set up a once a day re-index and then switch to a continuous index
update.
So everytime a content item is published it is immediately indexed and
deployed.  Problem persists with both scenario's.

We are using jconsole as a profiler but this does not give us information
if tomcat objects are released from their references and so are not garbage
collected.
Certainly its looks like the long-term heap memory is the issue and is not
being garbage collected effectively.

We dont even know what the likely culprit is.  i.e memory leak, heap/pergen
memory settings, lucene bug, bug in the way the code is interacting with
lucene.

We have been looking at this problem for months now..

We dont index xml files with xalan.

Kind Regards
Adrena Keating




                                                                                         
                                               
                      Ronnie Kolehmainen                                                 
                                               
                      <ronnie.kolehmaine         To:      java-user@lucene.apache.org 
                                                  
                      n@ub.uu.se>                cc:                                  
                                                  
                                                 Subject: Re: Lucene/Tomcat Memory Leak Issue
                                           
                      08/11/2006 06:23                                                   
                                               
                      PM                                                                 
                                               
                      Please respond to                                                  
                                               
                      java-user                                                          
                                               
                                                                                         
                                               




How do you index your documents? Are you releasing old resources? Can you
use a
profiler to see referenced objects?

I've experienced the same problem when indexing xml files which were parsed
with
xalan, and the memory leak in that case was in xalan. Switching to saxon
solved
the problems for us.

/Ronnie

Citerar adrena.keating@nl.abnamro.com:

>
> Hello, can anyone help?
>
>
> We're experiencing the following issue an Widows Intranet website:
>
>
> Following a Tomcat restart, our application has Lucene creating a single
> new index in a RAMDirectory, followed by continuous creation of
additional
> index entries as new content is published.
>
>
> During the initial index creation process, Heap Memory utilization jumps
to
> over 300 MB and never drops back down, even though the new index itself
is
> ~15 MB in size. Then, as additional index entries are added, the Heap
> Memory utilization continues to increase and garbage collection reduces
in
> size, until we max out the available Heap Memory and Tomcat crashes.
>
>
> Relevant configuration items:
>         Windows 2000 5.0
>         4 Processors, 2GB RAM
>         Tomcat 5.0.27
>         JDK 1.5.0_06
>         Lucene 1.9.1
>         VM arguments:
>                 -Djava.io.tmpdir=D:\Appldata\Lucene\lockDir
>                 -Dorg.apache.lucene.commitLockTimeout=60000
>                 -Dorg.apache.lucene.writeLockTimeout=60000
>                 -XX:PermSize=256m
>                 -XX:MaxPermSize=512m
>                 -Xrs
>                 -Xms256m
>                 -Xmx1024m
>                 -Xss512k
>         lucene.searcher.keep.open=true
>         lucene.searcher.retries=6
>
>
> Has anyone experienced similar conditions? Any suggestions?
>
> Kind Regards
> Adrena Keating
>
>
>
>
---------------------------------------------------------------------------
> This message (including any attachments) is confidential and may be
> privileged. If you have received it by mistake please notify the sender
by
> return e-mail and delete this message from your system. Any unauthorised
> use or dissemination of this message in whole or in part is strictly
> prohibited. Please note that e-mails are susceptible to change. ABN AMRO
> Bank N.V, which has its seat at Amsterdam, the Netherlands, and is
> registered in the Commercial Register under number 33002587, including
its
> group companies, shall not be liable for the improper or incomplete
> transmission of the information contained in this communication nor for
any
> delay in its receipt or damage to your system. ABN AMRO Bank N.V. (or its
> group companies) does not guarantee that the integrity of this
> communication has been maintained nor that this communication is free of
> viruses, interceptions or interference.
>
---------------------------------------------------------------------------
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org




---------------------------------------------------------------------------
This message (including any attachments) is confidential and may be
privileged. If you have received it by mistake please notify the sender by
return e-mail and delete this message from your system. Any unauthorised
use or dissemination of this message in whole or in part is strictly
prohibited. Please note that e-mails are susceptible to change. ABN AMRO
Bank N.V, which has its seat at Amsterdam, the Netherlands, and is
registered in the Commercial Register under number 33002587, including its
group companies, shall not be liable for the improper or incomplete
transmission of the information contained in this communication nor for any
delay in its receipt or damage to your system. ABN AMRO Bank N.V. (or its
group companies) does not guarantee that the integrity of this
communication has been maintained nor that this communication is free of
viruses, interceptions or interference.
---------------------------------------------------------------------------


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message