groovy-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Max Kuchin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GROOVY-7498) Native memory leak
Date Sat, 18 Jul 2015 04:14:04 GMT

    [ https://issues.apache.org/jira/browse/GROOVY-7498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632260#comment-14632260
] 

Max Kuchin commented on GROOVY-7498:
------------------------------------

Another example of VM.native_memory detail.diff, different web app, same environment. We made
a baseline at day 1, please find a detail.diff on day 2:

Native Memory Tracking:
 
Total:  reserved=7026506KB  +397530KB, committed=7026334KB +397550KB
 
-                 Java Heap (reserved=5447680KB, committed=5447680KB)
                            (mmap: reserved=5447680KB, committed=5447680KB)
 
-                     Class (reserved=6261KB +406KB, committed=6261KB +406KB)
                            (classes #21810 +230)
                            (malloc=6261KB +406KB, #73801 +1011)
 
-                    Thread (reserved=584KB -153KB, committed=584KB -153KB)
                            (thread #115 -30)
                            (stack: reserved=118220KB -30839KB, committed=118220KB -30839KB)
                            (malloc=363KB -96KB, #467 -120)
                            (arena=221KB -57KB, #230 -60)
 
-                      Code (reserved=58922KB +265KB, committed=58782KB +285KB)
                            (malloc=9002KB +265KB, #11430 +211)
                            (mmap: reserved=49920KB, committed=49780KB +20KB)
 
-                        GC (reserved=414375KB +21372KB, committed=414375KB +21372KB)
                            (malloc=212195KB +21372KB, #93189 +10399)
                            (mmap: reserved=202180KB, committed=202180KB)
 
-                  Compiler (reserved=382KB -140KB, committed=382KB -140KB)
                            (malloc=284KB -140KB, #402 -135)
                            (arena=98KB, #2)
 
-                  Internal (reserved=882616KB +392341KB, committed=882584KB +392341KB)
                            (malloc=882584KB +392341KB, #1358120 +542334)
                            (mmap: reserved=32KB, committed=0KB)
 
-                    Symbol (reserved=22877KB +17KB, committed=22877KB +17KB)
                            (malloc=18937KB +17KB, #200719 +345)
                            (arena=3940KB, #1)
 
-           Memory Tracking (reserved=74080KB +15007KB, committed=74080KB +15007KB)
                            (malloc=74080KB +15007KB, #456 -578)
 
-        Pooled Free Chunks (reserved=510KB -741KB, committed=510KB -741KB)
                            (malloc=510KB -741KB)
 
 
Details:
 
[0x00007ff6cc2428d8] CHeapObj<(unsigned short)1280>::operator new(unsigned long, unsigned
char*)+0x28
                            (malloc=8426KB +815KB, #82615 +9477)
 
[0x00007ff6cc24ae89] Arena::operator new(unsigned long, unsigned short)+0x39
                            (malloc=10KB -2KB, #233 -60)
 
[0x00007ff6cc24b9e3] ChunkPool::allocate(unsigned long, AllocFailStrategy::AllocFailEnum)+0x93
                            (malloc=4408KB -799KB, #369 -89)
 
[0x00007ff6cc24ba58] CHeapObj<(unsigned short)1792>::operator new(unsigned long, unsigned
char*)+0x28
                            (malloc=379537KB +158200KB, #1279421 +532818)
 
[0x00007ff6cc2bbdb9] ArrayAllocator<unsigned long, (unsigned short)1792>::allocate(unsigned
long)+0x49
                            (malloc=43610KB +4754KB, #74886 +9508)
 
[0x00007ff6cc3b6258] CHeapObj<(unsigned short)256>::operator new(unsigned long, unsigned
char*)+0x28
                            (malloc=1497KB +46KB, #70233 +716)
 
[0x00007ff6cc3c9469] CodeBlob::set_oop_maps(OopMapSet*)+0x49
                            (malloc=8834KB +252KB, #10047 +162)
 
[0x00007ff6cc3d08d8] CHeapObj<(unsigned short)1024>::operator new(unsigned long, unsigned
char*)+0x28
                            (malloc=97KB +13KB, #1146 +49)
 
[0x00007ff6cc3e7198] CHeapObj<(unsigned short)1536>::operator new(unsigned long, unsigned
char*)+0x28
                            (malloc=16KB -1KB, #178 -27)
 
[0x00007ff6cc44f0aa] Deoptimization::fetch_unroll_info_helper(JavaThread*)+0x9da
                            (malloc=152KB -74KB, #55 -27)
 
[0x00007ff6cc537bfd] GenericGrowableArray::raw_allocate(int)+0x9d
                            (malloc=459093KB +229383KB, #2623)
 
[0x00007ff6cc745bb6] MemBaseline::baseline_malloc_details(MemPointerArray const*)+0x1f6
                            (malloc=2KB +2KB, #1 +1)
 
[0x00007ff6cc7464f3] MemBaseline::baseline_vm_details(MemPointerArray const*)+0x493
                            (malloc=15KB +15KB, #1 +1)
 
[0x00007ff6cc747b20] MemPointerArrayImpl<VMCallsitePointer>::is_full()+0x40
                            (malloc=34KB +34KB, #1 +1)
 
[0x00007ff6cc7485a8] MemRecorder::MemRecorder()+0x208
                            (malloc=5241KB -7018KB, #436 -584)
 
[0x00007ff6cc74fbe4] MemPointerArrayImpl<MemPointerRecordEx>::is_full()+0x44
                            (malloc=67960KB +21640KB, #1)
 
[0x00007ff6cc74ffd6] MemPointerArrayImpl<SeqMemPointerRecordEx>::shrink()+0x86
                            (malloc=0KB -95KB, #0 -1)
 
[0x00007ff6cc750044] MemPointerArrayImpl<SeqMemPointerRecordEx>::is_full()+0x44
                            (malloc=672KB +432KB, #2 +1)
 
[0x00007ff6cc7c2a55] OopMapCache::OopMapCache()+0x45
                            (malloc=3198KB +360KB, #2558 +288)
 
[0x00007ff6cc7d1d88] CHeapObj<(unsigned short)512>::operator new(unsigned long, unsigned
char*)+0x28
                            (malloc=25KB -6KB, #116 -30)
 
[0x00007ff6cc7ef76b] ParkEvent::operator new(unsigned long)+0x3b
                            (malloc=311KB +4KB, #652 +8)
 
[0x00007ff6cc83a7cf] PtrQueueSet::allocate_buffer()+0xff
                            (malloc=13679KB +6507KB, #2771 +984)
 
[0x00007ff6cc8841e5] RSHashTable::RSHashTable(unsigned long)+0x85
                            (malloc=45793KB +12772KB, #2561 -31)
 
[0x00007ff6cc88422e] RSHashTable::RSHashTable(unsigned long)+0xce
                            (malloc=4579KB +1277KB, #2561 -31)
 
[0x00007ff6cc8c62f3] Symbol::operator new(unsigned long, int, Thread*)+0x53
                            (malloc=10461KB +17KB, #199730 +345)
 
[0x00007ff6cc9059b6] Thread::allocate(unsigned long, bool, unsigned short)+0x56
                            (malloc=328KB -87KB, #114 -30)
 
[0x00007ff6cc946e71] vframeArray::allocate(JavaThread*, int, GrowableArray<compiledVFrame*>*,
RegisterMap*, frame, frame, frame)+0x101
                            (malloc=112KB -60KB, #55 -27)
 
[0x00007ff6cc907034] JavaThread::run()+0x24
                            (mmap: reserved=102800KB -30839KB, committed=102800KB -30839KB)
 
[0x00007ff6cc94a536] ReservedSpace::initialize(unsigned long, unsigned long, bool, c+0x266
                            (mmap: reserved=11816KB, committed=11676KB +20KB)

At day 3, we can't get detail.diff and got an error "Native memory tracking has been shutdown
due to out of native memory".
We using large pages (-XX:+UseLargePages) so all java heap is in large memory and we can measure
native memory consumption looking at residential memory of the java process. It was around
400M after start and on 3rd day becomes 2.241G.


> Native memory leak
> ------------------
>
>                 Key: GROOVY-7498
>                 URL: https://issues.apache.org/jira/browse/GROOVY-7498
>             Project: Groovy
>          Issue Type: Bug
>    Affects Versions: 2.3.11
>         Environment: Ubuntu 14.04 , grails 2.4.5 java 1.7.0_80, tomcat 7.0.56
>            Reporter: Max Kuchin
>
> We are experiencing JVM native memory leak after upgrading our web application from grails
2.4.4 (groovy 2.3.7) to grails 2.4.5 (groovy 2.3.11).
> It disappears if we replace  groovy-all-2.3.11.jar in web application with groovy-all-2.3.7.jar.

> As a result of the leak JVM native memory usage can grow to 4G in two weeks after start
of web application.
> We'd enabled native memory tracking (-XX:NativeMemoryTracking=detail) to gather some
details. Here are results of native memory summary snapshot made with command jcmd pid VM.native_memory
summary
> Native memory summary after start of application:
> Total:  reserved=6039782KB,  committed=6039342KB
> -                 Java Heap (reserved=5447680KB, committed=5447680KB)
>                             (mmap: reserved=5447680KB, committed=5447680KB)
> -                     Class (reserved=2944KB, committed=2944KB)
>                             (classes #19490)
>                             (malloc=2944KB, #56242)
> -                    Thread (reserved=131202KB, committed=131202KB)
>                             (thread #127)
>                             (stack: reserved=130556KB, committed=130556KB)
>                             (malloc=402KB, #515)
>                             (arena=244KB, #254)
> -                      Code (reserved=54669KB, committed=54261KB)
>                             (malloc=4749KB, #7615)
>                             (mmap: reserved=49920KB, committed=49512KB)
> -                        GC (reserved=350261KB, committed=350261KB)
>                             (malloc=148081KB, #28742)
>                             (mmap: reserved=202180KB, committed=202180KB)
> -                  Compiler (reserved=399KB, committed=399KB)
>                             (malloc=301KB, #480)
>                             (arena=98KB, #2)
> -                  Internal (reserved=14366KB, committed=14334KB)
>                             (malloc=14334KB, #21616)
>                             (mmap: reserved=32KB, committed=0KB)
> -                    Symbol (reserved=20911KB, committed=20911KB)
>                             (malloc=17355KB, #180610)
>                             (arena=3556KB, #1)
> -           Memory Tracking (reserved=14986KB, committed=14986KB)
>                             (malloc=14986KB, #492)
> -        Pooled Free Chunks (reserved=2365KB, committed=2365KB)
>                             (malloc=2365KB)
> One day after start:
>  
> Total:  reserved=6610742KB , committed=6610518KB
>  -                 Java Heap (reserved=5447680KB, committed=5447680KB)
>                             (mmap: reserved=5447680KB, committed=5447680KB)
>  -                     Class (reserved=5525KB, committed=5525KB)
>                             (classes #21209)
>                             (malloc=5525KB, #71960)
> -                    Thread (reserved=646KB, committed=646KB)
>                             (thread #127)
>                             (stack: reserved=130556KB, committed=130556KB)
>                             (malloc=402KB +13KB, #515)
>                             (arena=244KB +8KB, #254)
> -                      Code (reserved=58182KB, committed=57990KB)
>                             (malloc=8262KB, #11074)
>                             (mmap: reserved=49920KB, committed=49728KB) 
> -                        GC (reserved=423906KB, committed=423906KB)
>                             (malloc=221726KB, #82681)
>                             (mmap: reserved=202180KB, committed=202180KB) 
> -                  Compiler (reserved=440KB, committed=440KB)
>                             (malloc=342KB +20KB, #513)
>                             (arena=98KB, #2)
> -                  Internal (reserved=472055KB, committed=472023KB)
>                             (malloc=472023KB, #746397)
>                             (mmap: reserved=32KB, committed=0KB)
> -                    Symbol (reserved=21668KB +7KB, committed=21668KB +7KB)
>                             (malloc=17760KB +7KB, #185311)
>                             (arena=3908KB, #1)
> -           Memory Tracking (reserved=49894KB, committed=49894KB)
>                             (malloc=49894KB +2464KB, #536)
> -        Pooled Free Chunks (reserved=191KB, committed=191KB)
>                             (malloc=191KB)
> There is substantial increase in Internal memory usage and object count:
> 14366KB, #21616 -> 472023KB, #746397
> Native memory summary, one day after start, same application, different groovy version
(groovy-all-2.3.7.jar):
> Total:  reserved=6098509KB,  committed=6099309KB
> -                 Java Heap (reserved=5447680KB, committed=5447680KB)
>                             (mmap: reserved=5447680KB, committed=5447680KB) 
> -                     Class (reserved=6263KB, committed=6263KB)
>                             (classes #21179)
>                             (malloc=6263KB, #71507) 
> -                    Thread (reserved=117771KB, committed=118799KB)
>                             (thread #114)
>                             (stack: reserved=117192KB, committed=118220KB)
>                             (malloc=360KB, #463)
>                             (arena=219KB, #228) 
> -                      Code (reserved=58131KB, committed=57935KB)
>                             (malloc=8211KB, #10858)
>                             (mmap: reserved=49920KB, committed=49724KB) 
> -                        GC (reserved=383041KB, committed=383041KB)
>                             (malloc=180861KB, #80128)
>                             (mmap: reserved=202180KB, committed=202180KB) 
> -                  Compiler (reserved=268KB, committed=268KB)
>                             (malloc=170KB, #313)
>                             (arena=98KB, #2) 
> -                  Internal (reserved=41336KB, committed=41304KB)
>                             (malloc=41304KB, #69639)
>                             (mmap: reserved=32KB, committed=0KB) 
> -                    Symbol (reserved=21696KB, committed=21696KB)
>                             (malloc=17788KB, #185505)
>                             (arena=3908KB, #1)
> -           Memory Tracking (reserved=22135KB, committed=22135KB)
>                             (malloc=22135KB, #432)
> -        Pooled Free Chunks (reserved=189KB, committed=189KB)
>                             (malloc=189KB)
> Internal memory consumption, one day after start, comparing groovy 2.3.7 to groovy 2.3.11:
> 41336KB, #69639 - groovy 2.3.7 
> 472023KB, #746397 - groovy 2.3.11, more than 10 times increase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message