ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tysli2016 <Tom.YS...@cityline.com.hk>
Subject Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Date Mon, 19 Dec 2016 03:49:30 GMT
> Are you sure that all of them are in this map? What is the size of rmtCfgs
map?

This map contains about 1/6 of the total CacheConfigurations, the size of
rmtCfg map is 227,306,792:

    Class Name                                                                           
       
| Shallow Heap | Retained Heap 
   
-----------------------------------------------------------------------------------------------------------------------------

                                                                                         
       
|              |               
    java.util.HashMap$Node[65536] @ 0xcd657f20                                           
       
|      262,160 |   227,306,744 
    '- table java.util.HashMap @ 0x8423c790                                              
       
|           48 |   227,306,792 
       '- rmtCfgs
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81a78dc8 |           56 |   227,306,848 


The Leak Suspects report shows there are 6 DynamicCacheDescriptors, each
size about 216MB:
https://drive.google.com/open?id=0BwY2dxDlRYhBbllWZ3pEMW1Tc00

All 6 DynamicCacheDescriptors exhibit the same pattern, they all contains a
whole lot of CacheConfigurations in a map, the numbers does add up:

    Class Name                                                                  
|    Objects |  Shallow Heap |    Retained Heap 
   
-----------------------------------------------------------------------------------------------------------------------------

                                                                                
|            |               |                 
    java.util.HashMap$Node[]                                                    
|     33,252 |    10,886,584 | >= 1,446,877,296 
    java.util.HashMap                                                           
|    205,140 |     9,846,720 | >= 1,446,860,856 
    java.util.HashMap$Node                                                      
|  1,116,555 |    35,729,760 | >= 1,441,725,448 
    org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor          
|          6 |           336 |   >= 681,000,984 
    org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor          
|          6 |           336 |   >= 680,977,904 
    org.apache.ignite.configuration.CacheConfiguration                          
|     88,421 |    25,465,248 |   >= 677,386,632 
    org.apache.ignite.configuration.CacheConfiguration                          
|     88,418 |    25,464,384 |   >= 677,363,648 



> Actually this map can be non-empty only on a node which is not fully
> started yet. Basically, when a new node joins a topology, it collects the
> configuration from all nodes for validation check, does the check and
> cleans the map. All this is part of the join process. Is this the case?
> How many nodes do you have? 

There are 2 client nodes associated with the heap dump.
But they should already joined the cluster, cause these 2 client nodes could
be found in the topology with the visor cli. Would these CacheConfiguration
objects created for disconnect process as well? because I found they lead to
a GridCacheProcessor.cachesOnDisconnect:

    Class Name                                                                           
                                                                           
| Shallow Heap | Retained Heap 
   
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                                                                                         
                                                                           
|              |               
    java.util.HashMap$Node[65536] @ 0xcd657f20                                           
                                                                           
|      262,160 |   227,306,744 
    '- table java.util.HashMap @ 0x8423c790                                              
                                                                           
|           48 |   227,306,792 
       '- rmtCfgs
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81a78dc8                                                                    
|           56 |   227,306,848 
          '- value java.util.HashMap$Node @ 0xe066c018                                   
                                                                           
|           32 |            32 
             '- [11] java.util.HashMap$Node[16] @ 0xe066bf48                             
                                                                           
|           80 |           272 
                '- table java.util.HashMap @ 0xe066bf08                                  
                                                                           
|           48 |           320 
                   '- cachesOnDisconnect
org.apache.ignite.internal.processors.cache.GridCacheProcessor @ 0x81bacb08              
                                  
|           80 |         2,640 



> To be honest, it's very hard to get to the bottom without a heap dump...
> Is it possible to reproduce in a standalone test without sensitive data?

Yes, we understand that fixes cannot be completely concluded without a
faithfully bug reproduction, we are trying to reproduce the symptom on
another machines. You know some OOMEs are so intermittent and quite time
consuming to reproduce them.
And we might also approach the problem the other way round, try to find what
might cause the symptom so that we might be able to reproduce it easier.

How does Ignite handle multiple connection event? 
Does Ignite also create CacheConfiguration on disconnect?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9623.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message