ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zbyszek <zbab...@yahoo.com>
Subject Memory leak in GridCachePartitionExchangeManager?
Date Mon, 18 Dec 2017 14:05:35 GMT

Dear all,

I was wondering if this is a know issue which has a chance to be fixed in
future or (I hope) it is me who missed something obvious in working with
Ignite caches.
I have a simple single node test app (built to investigate a memory leak
observed in our PROD deployment), that creates c.a. 20 LOCAL caches per sec.
with the config below:

private IgniteCache<String, BinaryObject> createLocalCache(String name) {
        CacheConfiguration<String, BinaryObject> cCfg = new
CacheConfiguration<>();
        cCfg.setName(name);
        cCfg.setGroupName("localCaches"); // without group leak is much
bigger!
        cCfg.setStoreKeepBinary(true);
        cCfg.setCacheMode(CacheMode.LOCAL);
        cCfg.setOnheapCacheEnabled(false);
        cCfg.setCopyOnRead(false);
        cCfg.setBackups(0);
        cCfg.setWriteBehindEnabled(false);
        cCfg.setReadThrough(false);
        cCfg.setReadFromBackup(false);
        cCfg.setQueryEntities(<some basic query here>);
        return ignite.createCache(cCfg).withKeepBinary();
    }

The caches are placed in the queue and are picked up by the worker thread
which just destorys them after removing from the queue. 
This setup seems to generate a memory leak of about 1GB per day.
When looking at heapdump, I see all space is occupied by instances of
java.util.concurrent.ConcurrentSkipListMap$Node:

(please copy paste table into notepad to see tables correctly)

Objects by class
+------------------------------------------------------------------------------------------------------------------------------------+--------------------+----------------------+---------------------------+
|                                                               Class                    
                                          
|      Objects       |     Shallow Size     |       Retained Size       |
+------------------------------------------------------------------------------------------------------------------------------------+--------------------+----------------------+---------------------------+
|  java.util.concurrent.ConcurrentSkipListMap$Node                                       
                                          
|  4,987,415   13 %  |  119,697,960   10 %  |  ~  1,204,893,605  100 %  |
| 
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask        
                                       
|  4,985,687   13 %  |  239,312,976   20 %  |    ~  917,361,000   76 %  |
| 
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner
                          
|  4,985,680   13 %  |  119,656,320   10 %  |    ~  558,390,752   46 %  |
|  org.jsr166.ConcurrentHashMap8                                                         
                                          
|  4,990,926   13 %  |  199,637,040   17 %  |    ~  439,459,352   36 %  |
|  org.jsr166.LongAdder8                                                                 
                                          
|  4,992,416   13 %  |  159,757,312   13 %  |    ~  159,757,312   13 %  |
|  org.apache.ignite.lang.IgniteUuid                                                     
                                          
|  4,989,306   13 %  |  119,743,344   10 %  |    ~  119,745,456   10 %  |
|  java.util.concurrent.ConcurrentSkipListMap$Index                                      
                                          
|  2,488,987    7 %  |   59,735,688    5 %  |    ~  119,502,384   10 %  |
|  java.util.concurrent.ConcurrentSkipListMap$HeadIndex                                  
                                          
|         49    0 %  |        1,568    0 %  |    ~  106,991,832    9 %  |
|  org.jsr166.ConcurrentHashMap8$ValuesView                                              
                                          
|  4,985,368   13 %  |   79,765,888    7 %  |     ~  79,765,888    7 %  |
|  java.util.HashMap$Node                                                                
                                          
|     44,335    0 %  |    1,418,720    0 %  |     ~  79,618,104    7 %  |
|  java.util.HashMap$Node[]                                                              
                                          
|     13,093    0 %  |    1,098,856    0 %  |     ~  68,150,520    6 %  |
|  java.util.HashMap                                                                     
                                          
|     13,550    0 %  |      650,400    0 %  |     ~  67,636,112    6 %  |
|  java.util.concurrent.ConcurrentSkipListMap                                            
                                          
|         10    0 %  |          480    0 %  |     ~  59,830,768    5 %  |


Merged paths to java.util.concurrent.ConcurrentSkipListMap$Node instances
(first 5 levels) reports no obvious dominator 
(at least no dominator from my test namespace):


Merged paths
+--------------------------------------------------------------------------------------------------------------------+--------------------+------------------------+--------------+
|                                                        Name                            
                          
|      Objects       |     Retained Size      |  Dominators  |
+--------------------------------------------------------------------------------------------------------------------+--------------------+------------------------+--------------+
|  +---<All the objects>                                                           
                                
|  4,987,415  100 %  |  1,037,112,344  100 %  |              |
|    |                                                                                   
                          
|                    |                        |              |
|    +---<Retained from several objects simultaneously (don't have a
dominator)>                                     |  1,245,699   25 %  | 
1,037,015,992   99 %  |              |
|    | |                                                                                 
                          
|                    |                        |              |
|    | +---java.util.concurrent.ConcurrentSkipListMap$Index                              
                          
|                    |                        |              |
|    | | |                                                                               
                          
|                    |                        |              |
|    | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                        
                          
|                    |                        |              |
|    | |   |                                                                             
                          
|                    |                        |              |
|    | |   +---java.util.concurrent.ConcurrentSkipListMap                                
                          
|                    |                        |              |
|    | |   | |                                                                           
                          
|                    |                        |              |
|    | |   | +---org.apache.ignite.internal.util.GridConcurrentSkipListSet               
                          
|                    |                        |              |
|    | |   |                                                                             
                          
|                    |                        |              |
|    | |   +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                      
                          
|                    |                        |              |
|    | |     |                                                                           
                          
|                    |                        |              |
|    | |     +---java.util.concurrent.ConcurrentSkipListMap                              
                          
|                    |                        |              |
|    | |     |                                                                           
                          
|                    |                        |              |
|    | |     +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                    
                          
|                    |                        |              |
|    | |                                                                                 
                          
|                    |                        |              |
|    | +---java.util.concurrent.ConcurrentSkipListMap$Node                               
                          
|                    |                        |              |
|    | | |                                                                               
                          
|                    |                        |              |
|    | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                        
                          
|                    |                        |              |
|    | |   |                                                                             
                          
|                    |                        |              |
|    | |   +---java.util.concurrent.ConcurrentSkipListMap                                
                          
|                    |                        |              |
|    | |     |                                                                           
                          
|                    |                        |              |
|    | |     +---org.apache.ignite.internal.util.GridConcurrentSkipListSet               
                          
|                    |                        |              |
|    | |                                                                                 
                          
|                    |                        |              |
|    | +---org.apache.ignite.thread.IgniteThread                                         
                          
|                    |                        |              |
|    |                                                                                   
                          
|                    |                        |              |
|    +---java.util.concurrent.ConcurrentSkipListMap$Node                                 
                          
|  3,741,257   75 %  |    777,983,576   75 %  |   3,741,257  |
|    | |                                                                                 
                          
|                    |                        |              |
|    | +---<Retained from several objects simultaneously (don't have a
dominator)>                                   |    934,092   19 %  |   
777,912,544   75 %  |              |
|    | | |                                                                               
                          
|                    |                        |              |
|    | | +---java.util.concurrent.ConcurrentSkipListMap$Index                            
                          
|                    |                        |              |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                      
                          
|                    |                        |              |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                    
                          
|                    |                        |              |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | | +---java.util.concurrent.ConcurrentSkipListMap$Index                          
                          
|                    |                        |              |
|    | | |   |                                                                           
                          
|                    |                        |              |
|    | | |   +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                    
                          
|                    |                        |              |
|    | | |   |                                                                           
                          
|                    |                        |              |
|    | | |   +---java.util.concurrent.ConcurrentSkipListMap$Index                        
                          
|                    |                        |              |
|    | | |                                                                               
                          
|                    |                        |              |
|    | | +---java.util.concurrent.ConcurrentSkipListMap$Node                             
                          
|                    |                        |              |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                      
                          
|                    |                        |              |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | | +---java.util.concurrent.ConcurrentSkipListMap                              
                          
|                    |                        |              |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | | +---java.util.concurrent.ConcurrentSkipListMap$Index                          
                          
|                    |                        |              |
|    | | |   |                                                                           
                          
|                    |                        |              |
|    | | |   +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                    
                          
|                    |                        |              |
|    | | |   |                                                                           
                          
|                    |                        |              |
|    | | |   +---java.util.concurrent.ConcurrentSkipListMap$Index                        
                          
|                    |                        |              |
|    | | |                                                                               
                          
|                    |                        |              |
|    | | +---org.apache.ignite.thread.IgniteThread                                       
                          
|                    |                        |              |
|    | |                                                                                 
                          
|                    |                        |              |
|    | +---java.util.concurrent.ConcurrentSkipListMap$Node                               
                          
|  2,806,816   56 %  |    583,674,344   56 %  |   2,806,816  |
|    | | |                                                                               
                          
|                    |                        |              |
|    | | +---<Retained from several objects simultaneously (don't have a
dominator)>                                 |    700,884   14 %  |   
583,622,824   56 %  |              |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | | +---java.util.concurrent.ConcurrentSkipListMap$Index                          
                          
|                    |                        |              |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                    
                          
|                    |                        |              |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index                        
                          
|                    |                        |              |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | | +---java.util.concurrent.ConcurrentSkipListMap$Node                           
                          
|                    |                        |              |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                    
                          
|                    |                        |              |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index                        
                          
|                    |                        |              |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node                         
                          
|                    |                        |              |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | | +---org.apache.ignite.thread.IgniteThread                                     
                          
|                    |                        |              |
|    | | |                                                                               
                          
|                    |                        |              |
|    | | +---java.util.concurrent.ConcurrentSkipListMap$Node                             
                          
|  2,105,683   42 %  |    437,877,768   42 %  |   2,105,683  |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | | +---<Retained from several objects simultaneously (don't have a
dominator)>                               |    525,764   11 %  |   
437,840,192   42 %  |              |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index                        
                          
|                    |                        |              |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node                         
                          
|                    |                        |              |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | | +---org.apache.ignite.thread.IgniteThread                                   
                          
|                    |                        |              |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | | +---java.util.concurrent.ConcurrentSkipListMap$Node                           
                          
|  1,579,741   32 %  |    328,509,568   32 %  |   1,579,741  |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | | +---<Retained from several objects simultaneously (don't have a
dominator)>                             |    394,396    8 %  |   
328,481,960   32 %  |              |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node                         
                          
|  1,185,216   24 %  |    246,468,448   24 %  |   1,185,216  |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                    
                          
|        129    0 %  |         27,608    0 %  |           2  |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                      
                          
|        178    0 %  |         37,576    0 %  |           2  |
|    | | |   |                                                                           
                          
|                    |                        |              |
|    | | |   +---java.util.concurrent.ConcurrentSkipListMap                              
                          
|        178    0 %  |         37,576    0 %  |           2  |
|    | | |                                                                               
                          
|                    |                        |              |
|    | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                        
                          
|        249    0 %  |         51,520    0 %  |           2  |
|    | |   |                                                                             
                          
|                    |                        |              |
|    | |   +---java.util.concurrent.ConcurrentSkipListMap                                
                          
|        249    0 %  |         51,520    0 %  |           2  |
|    | |     |                                                                           
                          
|                    |                        |              |
|    | |    
+---org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache           
             
|        249    0 %  |         51,520    0 %  |           2  |
|    | |                                                                                 
                          
|                    |                        |              |
|    | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                          
                          
|        348    0 %  |         71,008    0 %  |           2  |
|    | | |                                                                               
                          
|                    |                        |              |
|    | | +---java.util.concurrent.ConcurrentSkipListMap                                  
                          
|        348    0 %  |         71,008    0 %  |           2  |
|    | |   |                                                                             
                          
|                    |                        |              |
|    | |  
+---org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache           
               
|        348    0 %  |         71,008    0 %  |           2  |
|    | |     |                                                                           
                          
|                    |                        |              |
|    | |     +---<Retained from several objects simultaneously (don't have a
dominator)>                             |        348    0 %  |        
71,008    0 %  |              |
|    | |                                                                                 
                          
|                    |                        |              |
|    | +---java.util.concurrent.ConcurrentSkipListMap$KeyIterator                        
                          
|          1    0 %  |             24    0 %  |           1  |
|    |   |                                                                               
                          
|                    |                        |              |
|    |   +---org.apache.ignite.thread.IgniteThread                                       
                          
|          1    0 %  |             24    0 %  |           1  |
|    |                                                                                   
                          
|                    |                        |              |
|    +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex                            
                          
|        458    0 %  |         96,304    0 %  |          11  |
|    | |                                                                                 
                          
|                    |                        |              |
|    | +---java.util.concurrent.ConcurrentSkipListMap                                    
                          
|        457    0 %  |         96,280    0 %  |          10  |
|    | | |                                                                               
                          
|                    |                        |              |
|    | |
+---org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache           
                 
|        451    0 %  |         96,136    0 %  |           2  |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | | +---<Retained from several objects simultaneously (don't have a
dominator)>                               |        451    0 %  |        
96,136    0 %  |              |
|    | | |   |                                                                           
                          
|                    |                        |              |
|    | | |  
+---org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$CacheGroupHolder1
          
|                    |                        |              |
|    | | |   |                                                                           
                          
|                    |                        |              |
|    | | |  
+---org.apache.ignite.internal.processors.cache.CacheGroupContext                        
             
|                    |                        |              |
|    | | |   |                                                                           
                          
|                    |                        |              |
|    | | |  
+---org.apache.ignite.internal.processors.cache.GridCacheAffinityManager                 
             
|                    |                        |              |
|    | | |                                                                               
                          
|                    |                        |              |
|    | | +---org.apache.ignite.internal.util.GridBoundedConcurrentOrderedSet             
                          
|          4    0 %  |             96    0 %  |           4  |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | |
+---org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager   
               
|          4    0 %  |             96    0 %  |           2  |
|    | | |   |                                                                           
                          
|                    |                        |              |
|    | | |   +---<Retained from several objects simultaneously (don't have a
dominator)>                             |          4    0 %  |            
96    0 %  |              |
|    | | |                                                                               
                          
|                    |                        |              |
|    | | +---<Retained from several objects simultaneously (don't have a
dominator)>                                 |          1    0 %  |            
24    0 %  |              |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | | +---org.apache.ignite.internal.util.GridConcurrentSkipListSet                 
                          
|                    |                        |              |
|    | | | | |                                                                           
                          
|                    |                        |              |
|    | | | |
+---org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor                   
             
|                    |                        |              |
|    | | | |                                                                             
                          
|                    |                        |              |
|    | | | +---org.apache.ignite.thread.IgniteThread                                     
                          
|                    |                        |              |
|    | | |                                                                               
                          
|                    |                        |              |
|    | |
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager        
                 
|          1    0 %  |             24    0 %  |           1  |
|    | |   |                                                                             
                          
|                    |                        |              |
|    | |   +---<Retained from several objects simultaneously (don't have a
dominator)>                               |          1    0 %  |            
24    0 %  |              |
|    | |     |                                                                           
                          
|                    |                        |              |
|    | |     +---java.util.LinkedList$Node                                               
                          
|                    |                        |              |
|    | |     |                                                                           
                          
|                    |                        |              |
|    | |    
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$1      
             
|                    |                        |              |
|    | |     |                                                                           
                          
|                    |                        |              |
|    | |    
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2      
             
|                    |                        |              |
|    | |     |                                                                           
                          
|                    |                        |              |
|    | |    
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$3      
             
|                    |                        |              |
|    | |     |                                                                           
                          
|                    |                        |              |
|    | |    
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$4      
             
|                    |                        |              |
|    | |     |                                                                           
                          
|                    |                        |              |
|    | |    
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5      
             
|                    |                        |              |
|    | |     |                                                                           
                          
|                    |                        |              |
|    | |    
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker
      
|                    |                        |              |
|    | |     |                                                                           
                          
|                    |                        |              |
|    | |    
+---org.apache.ignite.internal.processors.cache.GridCacheSharedContext                   
             
|                    |                        |              |
|    | |                                                                                 
                          
|                    |                        |              |
|    | +---org.apache.ignite.internal.util.GridBoundedConcurrentOrderedMap               
                          
|          1    0 %  |             24    0 %  |           1  |
|    |   |                                                                               
                          
|                    |                        |              |
|    |  
+---org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager             
                 
|          1    0 %  |             24    0 %  |           1  |
|    |     |                                                                             
                          
|                    |                        |              |
|    |     +---<Retained from several objects simultaneously (don't have a
dominator)>                               |          1    0 %  |            
24    0 %  |              |
|    |       |                                                                           
                          
|                    |                        |              |
|    |       +---java.util.LinkedList$Node                                               
                          
|                    |                        |              |
|    |       |                                                                           
                          
|                    |                        |              |
|    |      
+---org.apache.ignite.internal.processors.cache.GridCacheSharedContext                   
             
|                    |                        |              |
|    |       |                                                                           
                          
|                    |                        |              |
|    |      
+---org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$2           
             
|                    |                        |              |
|    |       |                                                                           
                          
|                    |                        |              |
|    |      
+---org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$3           
             
|                    |                        |              |
|    |       |                                                                           
                          
|                    |                        |              |
|    |      
+---org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$DeadlockDetectionListener

|                    |                        |              |
|    |                                                                                   
                          
|                    |                        |              |
|    +---java.util.concurrent.ConcurrentSkipListMap$KeyIterator                          
                          
|          1    0 %  |             48    0 %  |           1  |
|      |                                                                                 
                          
|                    |                        |              |
|      +---org.apache.ignite.thread.IgniteThread                                         
                          
|          1    0 %  |             48    0 %  |           1  |
+--------------------------------------------------------------------------------------------------------------------+--------------------+------------------------+--------------+




Also 2 Bigges objects - Dominators are reported as:


Objects with biggest retained size
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+
|                                                                                 
Name                                                                                 
|    Retained Size    |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+
|  +---java.util.concurrent.ConcurrentSkipListMap [Stack Local] size =
4,985,687                                                                                
       
|  59,710,656    5 %  |
|  |                                                                                     
                                                                              
|                     |
| 
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager        
                                                                           
|  24,527,600    2 %  |
|  | |                                                                                   
                                                                              
|                     |
|  |
+---org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeFutureSet
                                                               
|  24,525,728    2 %  |
|  |   |                                                                                 
                                                                              
|                     |
|  |   +---java.util.LinkedList size = 1,000                                             
                                                                              
|  24,525,680    2 %  |


I have also noticed that during the test the most active thread is always
"exchange-worker" with allocation reported by visualvm on the level of
200MB/sec at the end of the test.
Please let me know if this is a know issue - and if not - if you would be
interested in simple repro app.

Thank you in advance,
zbyszek.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


Mime
View raw message