lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From roz dev <rozde...@gmail.com>
Subject Memory leak?? with CloseableThreadLocal with use of Snowball Filter
Date Tue, 31 Jul 2012 18:34:11 GMT
Hi All

I am using Solr 4 from trunk and using it with Tomcat 6. I am noticing that
when we are indexing lots of data with 16 concurrent threads, Heap grows
continuously. It remains high and ultimately most of the stuff ends up
being moved to Old Gen. Eventually, Old Gen also fills up and we start
getting into excessive GC problem.

I took a heap dump and found that most of the memory is consumed by
CloseableThreadLocal which is holding a WeakHashMap of Threads and its
state.

Most of the old gen is full with ThreadLocal eating up 3GB of heap and heap
dump shows that all such entries are using Snowball Filter. I looked into
LUCENE-3841 and verified that my version of SOLR 4 has that code.

So, I am wondering the reason for this memory leak - is it due to some
other bug with Solr/Lucene?

Here is a brief snapshot of HeapDump showing the problem

Class
Name
| Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------------------------------------------------------------------
*org.apache.solr.schema.IndexSchema$SolrIndexAnalyzer @
0x300c3eb28
|           24 | 3,885,213,072*
|- <class> class org.apache.solr.schema.IndexSchema$SolrIndexAnalyzer @
0x2f9753340                                           |            0
|             0
|- this$0 org.apache.solr.schema.IndexSchema @
0x300bf4048
|           96 |       276,704
*|- reuseStrategy org.apache.lucene.analysis.Analyzer$PerFieldReuseStrategy
@ 0x300c3eb40                                      |           16 |
3,885,208,728*
|  |- <class> class
org.apache.lucene.analysis.Analyzer$PerFieldReuseStrategy @
0x2f98368c0                                   |            0 |             0
|  |- storedValue org.apache.lucene.util.CloseableThreadLocal @
0x300c3eb50                                                   |
24 | 3,885,208,712
|  |  |- <class> class org.apache.lucene.util.CloseableThreadLocal @
0x2f9788918                                              |            8
|             8
|  |  |- t java.lang.ThreadLocal @
0x300c3eb68
|           16 |            16
|  |  |  '- <class> class java.lang.ThreadLocal @ 0x2f80f0868 System
Class                                                    |            8
|            24
*|  |  |- hardRefs java.util.WeakHashMap @
0x300c3eb78
|           48 | 3,885,208,656*
|  |  |  |- <class> class java.util.WeakHashMap @ 0x2f8476c00 System
Class                                                    |           16
|            16
|  |  |  |- table java.util.WeakHashMap$Entry[16] @
0x300c3eba8
|           80 | 2,200,016,960
|  |  |  |  |- <class> class java.util.WeakHashMap$Entry[] @
0x2f84789e8
|            0 |             0
|  |  |  |  |-* [7] java.util.WeakHashMap$Entry @
0x306a24950
|           40 |   318,502,920*
|  |  |  |  |  |- <class> class java.util.WeakHashMap$Entry @ 0x2f84786f8
System Class                                        |            0
|             0
|  |  |  |  |  |- queue java.lang.ref.ReferenceQueue @
0x300c3ebf8
|           32 |            48
|  |  |  |  |  |- referent java.lang.Thread @ 0x30678c2c0
web-23
|          112 |           160
|  |  |  |  |  |- value java.util.HashMap @
0x30678cbb0
|           48 |   318,502,880
|  |  |  |  |  |  |- <class> class java.util.HashMap @ 0x2f80b9428 System
Class                                               |           24
|            24
*|  |  |  |  |  |  |- table java.util.HashMap$Entry[32768] @
0x3c07c6f58                                                       |
131,088 |   318,502,832*
|  |  |  |  |  |  |  |- <class> class java.util.HashMap$Entry[] @
0x2f80bd9c8                                                 |            0
|             0
|  |  |  |  |  |  |  |- [10457] java.util.HashMap$Entry @
0x30678cbe0
|           32 |        40,864
|  |  |  |  |  |  |  |  |- <class> class java.util.HashMap$Entry @
0x2f80bd400 System Class                                   |            0
|             0
|  |  |  |  |  |  |  |  |- key java.lang.String @ 0x30678cc00
prod_desc_keywd_en_CA                                          |
32 |            96
|  |  |  |  |  |  |  |  |- value
org.apache.solr.analysis.TokenizerChain$SolrTokenStreamComponents @
0x30678cc60              |           24 |        20,344
|  |  |  |  |  |  |  |  |- next java.util.HashMap$Entry @
0x39a2c9100
|           32 |        20,392
|  |  |  |  |  |  |  |  |  |- <class> class java.util.HashMap$Entry @
0x2f80bd400 System Class                                |            0
|             0
|  |  |  |  |  |  |  |  |  |- key java.lang.String @ 0x39a2c9120
3637994_fr_CA_cat_name_keywd                                |           32
|           104
|  |  |  |  |  |  |  |  |  |- value
org.apache.solr.analysis.TokenizerChain$SolrTokenStreamComponents @
0x39a2c9188           |           24 |        20,256
|  |  |  |  |  |  |  |  |  |  |- <class> class
org.apache.solr.analysis.TokenizerChain$SolrTokenStreamComponents @
0x2f97a69a0|            0 |             0
|  |  |  |  |  |  |  |  |  |  |- this$0
org.apache.solr.analysis.TokenizerChain @
0x300bf6158                                 |           32 |        13,768
|  |  |  |  |  |  |  |  |  |  |- source
org.apache.lucene.analysis.core.WhitespaceTokenizer @
0x39a2c91a0                     |           64 |         8,304
|  |  |  |  |  |  |  |  | * |  |- sink
org.apache.lucene.analysis.snowball.SnowballFilter @
0x39a2c96a8                        |           48 |        10,736*
-------------------------------------------------------------------------------------------------------------------------------------------------------------



Any inputs are most welcome.

-Saroj

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message