lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Starzer (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (LUCENE-1186) [PATCH] Clear ThreadLocal instances in close()
Date Fri, 27 Feb 2009 11:54:12 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677281#action_12677281
] 

rviper edited comment on LUCENE-1186 at 2/27/09 3:53 AM:
-----------------------------------------------------------------

i'm using quartz schedules to trigger indexing tasks

since the analyzer class is using a thread local -> should i reuse an analyzer (e.g singleton
pattern) in this case (quartz job reusing threads (=tread pool) -> thread local data never
gets freed)...)? 

Mem dump fragment (after out of memory):

Class name Shallow Heap Retained HeapPercentage 
org.quartz.simpl.SimpleThreadPool$WorkerThread @ 0xb4e8070  QuartzScheduler_Worker-3 Thread
      120  55.914.144    21,65% 
|- java.lang.ThreadLocal$ThreadLocalMap @ 0xdf32a20      24  55.913.760  21,65% 
|  '- java.lang.ThreadLocal$ThreadLocalMap$Entry[16384] @ 0x163a61b8      65.552  55.913.736
21,65% 
java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xe090ca8  32 9.608 0,00% 
java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xe0f6b88     ....
java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xe5d3df8   ....
  org.apache.lucene.analysis.StopAnalyzer$SavedStreams @ 0xe090cc8  24      9.560    0,00%

  org.apache.lucene.analysis.LowerCaseTokenizer @ 0xe090ce0           32   8.520    0,00%

  org.apache.lucene.analysis.CharArraySet @ 0xe090e30       24          968   0,00% 
  org.apache.lucene.analysis.StopFilter @ 0xe090e18             24         24    0,00% 
  org.apache.lucene.analysis.StopAnalyzer @ 0xe0911f8          24          24     0,00%
java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xe629d80 ....
java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xeb53510 ....
......
...... 

quartz overview (eclipse memory analysis):

Class name    Shallow Heap | Retained Heap | Percentage 
org.quartz.simpl.SimpleThreadPool$WorkerThread @ 0xb4e8070  QuartzScheduler_Worker-3 Thread|
         120 |    55.914.144 |     21,65% 
org.quartz.simpl.SimpleThreadPool$WorkerThread @ 0xb4e7a50  QuartzScheduler_Worker-7 Thread|
         120 |    30.684.056 |     11,88% 
org.quartz.simpl.SimpleThreadPool$WorkerThread @ 0xb4e7710  QuartzScheduler_Worker-9 Thread|
         120 |    19.464.024 |      7,54% 
org.quartz.simpl.SimpleThreadPool$WorkerThread @ 0xb4e7ee8  QuartzScheduler_Worker-4 Thread|
         120 |    14.813.640 |      5,74% 
org.quartz.simpl.SimpleThreadPool$WorkerThread @ 0xb4e78c8  QuartzScheduler_Worker-8 Thread|
         120 |    11.154.576 |      4,32% 
org.quartz.simpl.SimpleThreadPool$WorkerThread @ 0xb4e81f8  QuartzScheduler_Worker-2 Thread|
         120 |     8.403.544 |      3,25% 
org.quartz.simpl.SimpleThreadPool$WorkerThread @ 0xb4e8380  QuartzScheduler_Worker-1 Thread|
         120 |     8.334.552 |      3,23% 
org.quartz.simpl.SimpleThreadPool$WorkerThread @ 0xb4e7d60  QuartzScheduler_Worker-5 Thread|
         120 |     8.314.904 |      3,22% 
org.quartz.simpl.SimpleThreadPool$WorkerThread @ 0xb4e8508  QuartzScheduler_Worker-0 Thread|
         120 |     8.267.376 |      3,20% 



      was (Author: rviper):
    i'm using quartz schedules to trigger indexing tasks

since the analyzer class is using a thread local -> should i reuse an analyzer (e.g singleton
pattern) in this case (quartz job reusing threads (=tread pool) -> thread local data never
gets freed)...)? 

Mem dump (part):

Class name Shallow Heap Retained HeapPercentage 

org.quartz.simpl.SimpleThreadPool$WorkerThread @ 0xb4e8070  QuartzScheduler_Worker-3 Thread
      120  55.914.144    21,65% 
|- java.lang.ThreadLocal$ThreadLocalMap @ 0xdf32a20      24  55.913.760  21,65% 
|  '- java.lang.ThreadLocal$ThreadLocalMap$Entry[16384] @ 0x163a61b8      65.552  55.913.736
21,65% 
java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xe090ca8  32 9.608 0,00% 
java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xe0f6b88     ....
java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xe5d3df8   ....
java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xe629d80 ....
java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xeb53510 ....
......
...... 


  
> [PATCH] Clear ThreadLocal instances in close()
> ----------------------------------------------
>
>                 Key: LUCENE-1186
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1186
>             Project: Lucene - Java
>          Issue Type: Improvement
>    Affects Versions: 2.3, 2.3.1, 2.4
>         Environment: any
>            Reporter: Christian Kohlschütter
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.4.1, 2.9
>
>         Attachments: LUCENE-1186-SegmentReader.patch, LUCENE-1186.patch
>
>
> As already found out in LUCENE-436, there seems to be a garbage collection problem with
ThreadLocals at certain constellations, resulting in an OutOfMemoryError.
> The resolution there was to remove the reference to the ThreadLocal value when calling
the close() method of the affected classes (see FieldsReader and TermInfosReader).
> For Java < 5.0, this can effectively be done by calling threadLocal.set(null); for
Java >= 5.0, we would call threadLocal.remove()
> Analogously, this should be done in *any* class which creates ThreadLocal values
> Right now, two classes of the core API make use of ThreadLocals, but do not properly
remove their references to the ThreadLocal value
> 1. org.apache.lucene.index.SegmentReader
> 2. org.apache.lucene.analysis.Analyzer
> For SegmentReader, I have attached a simple patch.
> For Analyzer, there currently is no patch because Analyzer does not provide a close()
method (future to-do?)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message