lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kieran <kie...@kitsite.com>
Subject Re: Lucene/Tomcat Memory Leak Issue
Date Thu, 17 Aug 2006 12:35:52 GMT
The source code is there to look at, modify, and/or compile yourself ;-)
Any "grave concerns" should be addressable by examining the contents of 
the patch.
The change is pretty minor. As far as problems are concerned: in some 
circumstances, there could (arguably) be a performance overhead.
We use it in a production environment; an array of 6 sun blade servers 
that was running out of memory regularly. This patch stopped this bug.
Our particular installation uses the RAMDirectory class to load indexes. 
This appears to vastly speed up the occurrence of the out of memory 
exceptions, but I don't think it's crucial.
If you want any more details, you're welcome to email me either on or 
off-list.
Kieran


adrena.keating@nl.abnamro.com wrote:
> Thanks Kieran,
>
> Yep iv seen this patch before but our organisation has grave concerns
> putting it into a production environment as its not verified/support by
> Lucene as an official patch nor have they verified that there is an actual
> bug.
>
> Do you know if you or anyone has actuall installed in prod env and where
> there any problems?
>
> Kind Regards
> Adrena Keating
>
>
>
>                                                                                     
                                                  
>                       kieran                                                        
                                                  
>                       <kieran@kitsite.         To:      java-user@lucene.apache.org
                                                   
>                       com>                     cc:                               
                                                     
>                                                Subject: Re: Lucene/Tomcat Memory Leak
Issue                                            
>                       08/17/2006 12:00                                              
                                                  
>                       PM                                                            
                                                  
>                       Please respond                                                
                                                  
>                       to java-user                                                  
                                                  
>                                                                                     
                                                  
>
>
>
>
> Hello Adrena,
>
> You might want to consult the following thread:
>
>   http://issues.apache.org/jira/browse/LUCENE-436
>
> The issue /might/ be to do with your JVM's implementation of ThreadLocal.
> The above discussion contains a fix/workaround for this issue.
>
> Kieran
>
>
>
> adrena.keating@nl.abnamro.com wrote:
>   
>> We are indexing the file server via Lucene.  We have a 15 MB index file
>>     
> and
>   
>> have set up a once a day re-index and then switch to a continuous index
>> update.
>> So everytime a content item is published it is immediately indexed and
>> deployed.  Problem persists with both scenario's.
>>
>> We are using jconsole as a profiler but this does not give us information
>> if tomcat objects are released from their references and so are not
>>     
> garbage
>   
>> collected.
>> Certainly its looks like the long-term heap memory is the issue and is
>>     
> not
>   
>> being garbage collected effectively.
>>
>> We dont even know what the likely culprit is.  i.e memory leak,
>>     
> heap/pergen
>   
>> memory settings, lucene bug, bug in the way the code is interacting with
>> lucene.
>>
>> We have been looking at this problem for months now..
>>
>> We dont index xml files with xalan.
>>
>> Kind Regards
>> Adrena Keating
>>
>>
>>
>>
>>
>>     
>
>   
>>                       Ronnie Kolehmainen
>>     
>
>   
>>                       <ronnie.kolehmaine         To:
>>     
> java-user@lucene.apache.org
>
>   
>>                       n@ub.uu.se>                cc:
>>     
>
>   
>>                                                  Subject: Re:
>>     
> Lucene/Tomcat Memory Leak Issue
>   
>>                       08/11/2006 06:23
>>     
>
>   
>>                       PM
>>     
>
>   
>>                       Please respond to
>>     
>
>   
>>                       java-user
>>     
>
>   
>
>   
>>
>>
>> How do you index your documents? Are you releasing old resources? Can you
>> use a
>> profiler to see referenced objects?
>>
>> I've experienced the same problem when indexing xml files which were
>>     
> parsed
>   
>> with
>> xalan, and the memory leak in that case was in xalan. Switching to saxon
>> solved
>> the problems for us.
>>
>> /Ronnie
>>
>> Citerar adrena.keating@nl.abnamro.com:
>>
>>
>>     
>>> Hello, can anyone help?
>>>
>>>
>>> We're experiencing the following issue an Widows Intranet website:
>>>
>>>
>>> Following a Tomcat restart, our application has Lucene creating a single
>>> new index in a RAMDirectory, followed by continuous creation of
>>>
>>>       
>> additional
>>
>>     
>>> index entries as new content is published.
>>>
>>>
>>> During the initial index creation process, Heap Memory utilization jumps
>>>
>>>       
>> to
>>
>>     
>>> over 300 MB and never drops back down, even though the new index itself
>>>
>>>       
>> is
>>
>>     
>>> ~15 MB in size. Then, as additional index entries are added, the Heap
>>> Memory utilization continues to increase and garbage collection reduces
>>>
>>>       
>> in
>>
>>     
>>> size, until we max out the available Heap Memory and Tomcat crashes.
>>>
>>>
>>> Relevant configuration items:
>>>         Windows 2000 5.0
>>>         4 Processors, 2GB RAM
>>>         Tomcat 5.0.27
>>>         JDK 1.5.0_06
>>>         Lucene 1.9.1
>>>         VM arguments:
>>>                 -Djava.io.tmpdir=D:\Appldata\Lucene\lockDir
>>>                 -Dorg.apache.lucene.commitLockTimeout=60000
>>>                 -Dorg.apache.lucene.writeLockTimeout=60000
>>>                 -XX:PermSize=256m
>>>                 -XX:MaxPermSize=512m
>>>                 -Xrs
>>>                 -Xms256m
>>>                 -Xmx1024m
>>>                 -Xss512k
>>>         lucene.searcher.keep.open=true
>>>         lucene.searcher.retries=6
>>>
>>>
>>> Has anyone experienced similar conditions? Any suggestions?
>>>
>>> Kind Regards
>>> Adrena Keating
>>>
>>>
>>>
>>>
>>>
>>>       
> ---------------------------------------------------------------------------
>   
>>> This message (including any attachments) is confidential and may be
>>> privileged. If you have received it by mistake please notify the sender
>>>
>>>       
>> by
>>
>>     
>>> return e-mail and delete this message from your system. Any unauthorised
>>> use or dissemination of this message in whole or in part is strictly
>>> prohibited. Please note that e-mails are susceptible to change. ABN AMRO
>>> Bank N.V, which has its seat at Amsterdam, the Netherlands, and is
>>> registered in the Commercial Register under number 33002587, including
>>>
>>>       
>> its
>>
>>     
>>> group companies, shall not be liable for the improper or incomplete
>>> transmission of the information contained in this communication nor for
>>>
>>>       
>> any
>>
>>     
>>> delay in its receipt or damage to your system. ABN AMRO Bank N.V. (or
>>>       
> its
>   
>>> group companies) does not guarantee that the integrity of this
>>> communication has been maintained nor that this communication is free of
>>> viruses, interceptions or interference.
>>>
>>>
>>>       
> ---------------------------------------------------------------------------
>   
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>       
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>>
>>     
> ---------------------------------------------------------------------------
>   
>> This message (including any attachments) is confidential and may be
>> privileged. If you have received it by mistake please notify the sender
>>     
> by
>   
>> return e-mail and delete this message from your system. Any unauthorised
>> use or dissemination of this message in whole or in part is strictly
>> prohibited. Please note that e-mails are susceptible to change. ABN AMRO
>> Bank N.V, which has its seat at Amsterdam, the Netherlands, and is
>> registered in the Commercial Register under number 33002587, including
>>     
> its
>   
>> group companies, shall not be liable for the improper or incomplete
>> transmission of the information contained in this communication nor for
>>     
> any
>   
>> delay in its receipt or damage to your system. ABN AMRO Bank N.V. (or its
>> group companies) does not guarantee that the integrity of this
>> communication has been maintained nor that this communication is free of
>> viruses, interceptions or interference.
>>
>>     
> ---------------------------------------------------------------------------
>   
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>     
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------------
> This message (including any attachments) is confidential and may be
> privileged. If you have received it by mistake please notify the sender by
> return e-mail and delete this message from your system. Any unauthorised
> use or dissemination of this message in whole or in part is strictly
> prohibited. Please note that e-mails are susceptible to change. ABN AMRO
> Bank N.V, which has its seat at Amsterdam, the Netherlands, and is
> registered in the Commercial Register under number 33002587, including its
> group companies, shall not be liable for the improper or incomplete
> transmission of the information contained in this communication nor for any
> delay in its receipt or damage to your system. ABN AMRO Bank N.V. (or its
> group companies) does not guarantee that the integrity of this
> communication has been maintained nor that this communication is free of
> viruses, interceptions or interference.
> ---------------------------------------------------------------------------
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>   



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message