ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tysli2016 <Tom.YS...@cityline.com.hk>
Subject Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Date Thu, 08 Dec 2016 03:47:45 GMT
We have 2 machines (M1, M2)
In M1 running a Ignite server (I1) and a Tomcat server (T1), which hosted 2
Ignite clients (C1a, C1b),
similarly in M2 running a Ignite server (I2) and a Tomcat server (T2), which
hosted 2 Ignite clients (C2a, C2b).

OutOfMemoryError were found in both T1 and T2 yesterday, we have a heap dump
from T1, but failed to get a heap dump from T2.

After the error, by using visor we found only T1, C1a, and C1b were still in
the cluster, all Ignite nodes on M2 were disconnected.

We've lost the Tomcat server log due to some reason.

>From the heap dump of T1 we found 6
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor holding
total 1.2GB memory:

    Class Name                                                                           
       
| Shallow Heap | Retained Heap | Percentage
   
------------------------------------------------------------------------------------------------------------------------------------------
                                                                                         
       
|              |               |           
    org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81a78dc8               |           56 |   227,306,848 |     14.05%
    '- java.util.HashMap @ 0x8423c790                                                    
       
|           48 |   227,306,792 |     14.05%
       '- java.util.HashMap$Node[65536] @ 0xcd657f20                                     
       
|      262,160 |   227,306,744 |     14.05%
    org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x8449bed0               |           56 |   227,299,144 |     14.05%
    org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81ae5c10               |           56 |   227,071,080 |     14.04%
    org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x8449f158               |           56 |   227,063,384 |     14.04%
    org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81adfa38               |           56 |   226,599,544 |     14.01%
    org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x8449f090               |           56 |   226,591,864 |     14.01%
    org.apache.catalina.loader.WebappClassLoader @ 0x80167cd8                            
       
|          136 |    29,521,360 |      1.83%
    org.apache.catalina.loader.WebappClassLoader @ 0x8363a078                            
       
|          136 |    28,471,928 |      1.76%
    com.cityline.cps.admin.api.controller.LoginController @ 0x819c54d0                   
       
|           48 |    21,042,952 |      1.30%
    org.apache.ignite.spi.discovery.tcp.ClientImpl @ 0x84477b20                          
       
|           96 |    14,456,360 |      0.89%
    org.apache.ignite.internal.processors.cache.GridCacheContext @
0xade21488                     |          152 |    10,462,696 |      0.65%
    org.apache.ignite.internal.processors.cache.GridCacheContext @
0xae8ec420                     |          152 |     9,800,264 |      0.61%
    org.apache.ignite.internal.processors.cache.GridCacheContext @
0x86aa66a0                     |          152 |     9,677,848 |      0.60%
    org.apache.ignite.internal.processors.cache.GridCacheContext @
0x86ac6b58                     |          152 |     9,634,288 |      0.60%
    org.apache.ignite.internal.processors.cache.GridCacheContext @
0x83013720                     |          152 |     6,075,032 |      0.38%
    org.apache.ignite.internal.processors.cache.GridCacheContext @
0x86008f08                     |          152 |     5,967,304 |      0.37%
    class java.beans.ThreadGroupContext @ 0x81070c30 System Class                        
       
|            8 |     2,634,888 |      0.16%
    org.hibernate.internal.SessionFactoryImpl @ 0x811ec9f0                               
       
|          136 |     2,406,176 |      0.15%
    org.hibernate.internal.SessionFactoryImpl @ 0x85de4c98                               
       
|          136 |     2,082,128 |      0.13%
    org.hibernate.internal.SessionFactoryImpl @ 0x83b30890                               
       
|          136 |     2,036,976 |      0.13%
    org.apache.ignite.internal.GridKernalContextImpl @ 0x841d4090                        
       
|          248 |     1,599,680 |      0.10%
    org.apache.ignite.internal.GridKernalContextImpl @ 0x818f6900                        
       
|          248 |     1,596,328 |      0.10%
    java.net.URLClassLoader @ 0x800fe890                                                 
       
|           80 |     1,451,640 |      0.09%
    org.apache.catalina.webresources.JarResourceSet @ 0x8019d4d0                         
       
|           72 |     1,302,400 |      0.08%
    org.apache.ignite.internal.processors.cache.GridCacheSharedContext @
0x8427bbc0               |           72 |     1,242,936 |      0.08%
    org.apache.ignite.internal.processors.cache.GridCacheSharedContext @
0x819c5e18               |           72 |     1,239,464 |      0.08%
    org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager
@ 0x819d54e8         |           88 |     1,052,272 |      0.07%
    org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager
@ 0x842cc828         |           88 |     1,052,272 |      0.07%
    org.apache.tomcat.util.net.NioEndpoint @ 0x80780e38                                  
       
|          288 |     1,052,136 |      0.07%
    org.apache.tomcat.util.net.NioEndpoint @ 0x80666fb0                                  
       
|          288 |     1,051,584 |      0.07%
    org.springframework.beans.factory.support.DefaultListableBeanFactory @
0x80fe9e50             |          208 |       882,424 |      0.05%
    org.apache.ignite.internal.processors.cache.GridCacheContext @
0x82fb0d28                     |          152 |       847,280 |      0.05%
    org.apache.ignite.internal.processors.cache.GridCacheContext @
0x85fa7cd0                     |          152 |       834,120 |      0.05%
    org.springframework.beans.factory.support.DefaultListableBeanFactory @
0x857e33d0             |          208 |       725,632 |      0.04%
    org.springframework.beans.factory.support.DefaultListableBeanFactory @
0x83b2fbb8             |          208 |       695,544 |      0.04%
   
org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource
@ 0x81367d80|           24 |       583,376 |      0.04%
    Total: 36 of 252,650 entries; 252,614 more                                           
       
|              |               |           
   
------------------------------------------------------------------------------------------------------------------------------------------



drilling into one of DynamicCacheDescriptor, found the memory was held by a
HashMap (java.util.HashMap @ 0x8423c790, java.util.HashMap$Node[65536] @
0xcd657f20), which seems to contain a bunch of CacheConfiguration objects:

    Class Name                                                                           
                        
| Shallow Heap | Retained Heap | Percentage
   
-----------------------------------------------------------------------------------------------------------------------------------------------------------
                                                                                         
                        
|              |               |           
    java.util.HashMap$Node[65536] @ 0xcd657f20                                           
                        
|      262,160 |   227,306,744 |     14.05%
    |- java.util.HashMap$Node @ 0xc559b860                                               
                        
|           32 |        38,520 |      0.00%
    |  |- java.util.HashMap$Node @ 0xcad62418                                            
                        
|           32 |        30,816 |      0.00%
    |  |  |- java.util.HashMap$Node @ 0xcb6d56b8                                         
                        
|           32 |        23,112 |      0.00%
    |  |  |  |- java.util.HashMap$Node @ 0xce1be088                                      
                        
|           32 |        15,408 |      0.00%
    |  |  |  |  |- java.util.HashMap$Node @ 0xd4e3dbe8                                   
                        
|           32 |         7,704 |      0.00%
    |  |  |  |  |  '- org.apache.ignite.configuration.CacheConfiguration @
0xd4e3be10                              |          288 |         7,672 |     
0.00%
    |  |  |  |  |     |-
org.apache.ignite.internal.processors.cache.GridCacheDefaultAffinityKeyMapper
@ 0xd4e3bfe0|           24 |         7,096 |      0.00%
    |  |  |  |  |     |- java.lang.String @ 0xd4e38260 
ignite-marshaller-sys-cache                                |           24 |           
96 |      0.00%
    |  |  |  |  |     |-
org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction @
0xd4e3bf90       |           40 |            80 |      0.00%
    |  |  |  |  |     |- java.util.HashSet @ 0xd4e3bf50                                  
                        
|           16 |            64 |      0.00%
    |  |  |  |  |     |-
javax.cache.configuration.FactoryBuilder$SingletonFactory @ 0xd4e3bf30                   
|           16 |            32 |      0.00%
    |  |  |  |  |     |-
org.apache.ignite.plugin.CachePluginConfiguration[0] @ 0xd4e3db98                        
|           16 |            16 |      0.00%
    |  |  |  |  |     '- Total: 6 entries                                                
                        
|              |               |           
    |  |  |  |  |- org.apache.ignite.configuration.CacheConfiguration @
0xce1be0a8                                 |          288 |         7,672 |     
0.00%
    |  |  |  |  |  |-
org.apache.ignite.internal.processors.cache.GridCacheDefaultAffinityKeyMapper
@ 0xce1be2e8   |           24 |         7,096 |      0.00%
    |  |  |  |  |  |- java.lang.String @ 0xce1be228 
ignite-marshaller-sys-cache                                   |           24
|            96 |      0.00%
    |  |  |  |  |  |-
org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction @
0xce1be298          |           40 |            80 |      0.00%
    |  |  |  |  |  |- java.util.HashSet @ 0xce1be1c8                                     
                        
|           16 |            64 |      0.00%
    |  |  |  |  |  |-
javax.cache.configuration.FactoryBuilder$SingletonFactory @ 0xce1be208                   
  
|           16 |            32 |      0.00%
    |  |  |  |  |  |- org.apache.ignite.plugin.CachePluginConfiguration[0] @
0xce1c2098                            |           16 |            16 |     
0.00%
    |  |  |  |  |  '- Total: 6 entries                                                   
                        
|              |               |           
    |  |  |  |  '- Total: 2 entries                                                      
                        
|              |               |           
    |  |  |  |- org.apache.ignite.configuration.CacheConfiguration @
0xcb6d56d8                                    |          288 |         7,672
|      0.00%
    |  |  |  '- Total: 2 entries                                                         
                        
|              |               |           
    |  |  |- org.apache.ignite.configuration.CacheConfiguration @ 0xcad62438             
                        
|          288 |         7,672 |      0.00%
    |  |  '- Total: 2 entries                                                            
                        
|              |               |           
    |  |- org.apache.ignite.configuration.CacheConfiguration @ 0xc559b880                
                        
|          288 |         7,672 |      0.00%
    |  '- Total: 2 entries                                                               
                        
|              |               |           
    |- java.util.HashMap$Node @ 0x8e45adf0                                               
                        
|           32 |        38,520 |      0.00%
    |- java.util.HashMap$Node @ 0x983d3e70                                               
                        
|           32 |        38,520 |      0.00%
    |- java.util.HashMap$Node @ 0x87b1c1c8                                               
                        
|           32 |        38,520 |      0.00%
    |- java.util.HashMap$Node @ 0x8a0d6980                                               
                        
|           32 |        38,520 |      0.00%
    |- java.util.HashMap$Node @ 0x97143f50                                               
                        
|           32 |        38,520 |      0.00%
    |- java.util.HashMap$Node @ 0x9731b858                                               
                        
|           32 |        38,520 |      0.00%
    |- java.util.HashMap$Node @ 0x906efbd8                                               
                        
|           32 |        38,520 |      0.00%
    |- java.util.HashMap$Node @ 0x9735be90                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x8ebdf958                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x951d6be0                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x9061a5a8                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x97b80fa0                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x877bad58                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x873bb2d0                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x8d47e168                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x89e19d58                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x8711c5f0                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x980292b0                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0xb25e7ac8                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x9d3ddbe0                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x948dd4d0                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x8d7eb430                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0xc60fa120                                               
                        
|           32 |        30,816 |      0.00%
    |- java.util.HashMap$Node @ 0x9cd18138                                               
                        
|           32 |        30,816 |      0.00%
    '- Total: 25 of 23,850 entries; 23,825 more                                          
                        
|              |               |           
   
-----------------------------------------------------------------------------------------------------------------------------------------------------------


and this is the shortest paths to the accumulation point for
java.util.HashMap @ 0x8423c790

    Class Name                                                                           
                                                                           
| Shallow Heap | Retained Heap
   
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
                                                                                         
                                                                           
|              |              
    java.util.HashMap$Node[65536] @ 0xcd657f20                                           
                                                                           
|      262,160 |   227,306,744
    '- table java.util.HashMap @ 0x8423c790                                              
                                                                           
|           48 |   227,306,792
       '- rmtCfgs
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81a78dc8                                                                    
|           56 |   227,306,848
          '- value java.util.HashMap$Node @ 0xe066c018                                   
                                                                           
|           32 |            32
             '- [11] java.util.HashMap$Node[16] @ 0xe066bf48                             
                                                                           
|           80 |           272
                '- table java.util.HashMap @ 0xe066bf08                                  
                                                                           
|           48 |           320
                   '- cachesOnDisconnect
org.apache.ignite.internal.processors.cache.GridCacheProcessor @ 0x81bacb08              
                                  
|           80 |         2,640
                      '- cacheProc
org.apache.ignite.internal.GridKernalContextImpl @ 0x818f6900                            
                                        
|          248 |     1,596,328
                         |- ctx org.apache.ignite.internal.IgniteKernal @
0x818f68a0                                                                               
 
|           96 |           200
                         |  |- ignite
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi @ 0x819c19c0                 
                                     
|          240 |         2,000
                         |  |  |- this$0
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker
@ 0x81b97128  tcp-comm-worker-#1%null% Thread|          144 |         2,640
                         |  |  |- this$0
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$3 @ 0x81b97098               
                                  
|           16 |            16
                         |  |  '- Total: 2 entries                                       
                                                                           
|              |              
                         |  |- ignite
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi @ 0x81968500                         
                                     
|          256 |       104,176
                         |  |- grid
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance @ 0x81967d28                   
                                       
|           88 |           344
                         |  '- Total: 3 entries                                          
                                                                           
|              |              
                         |- ctx
org.apache.ignite.internal.processors.clock.GridClockServer @ 0x819680e0                 
                                           
|           32 |            32
                         |- ctx
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager @
0x81968468                                                       |         
128 |        45,944
                         |- ctx
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor @
0x81b97c98                                                       |          
32 |           512
                         |- ctx
org.apache.ignite.internal.processors.rest.GridRestProcessor @ 0x81bad1a0                
                                           
|           64 |         1,056
                         |- ctx
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor @
0x83009a50                                                   |           48
|         2,008
                         |- ctx
org.apache.ignite.internal.managers.deployment.GridDeploymentManager @
0x818f69f8                                                     |          
48 |         2,496
                         '- Total: 7 entries                                             
                                                                           
|              |              
   
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message