[ https://issues.apache.org/jira/browse/GEODE-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961471#comment-16961471
]
Kirk Lund edited comment on GEODE-7370 at 10/28/19 9:46 PM:
------------------------------------------------------------
Here are some ClassGraph issues (all have been fixed and are included in 4.8.52) which may
have contributed to memory issues involving older releases (of ClassGraph):
Memory leak of ScanResults #233
https://github.com/classgraph/classgraph/issues/233
ScanResult leaks shutdown hooks #240
https://github.com/classgraph/classgraph/issues/240
Path scanning does deep scans for whitelisted paths #282
https://github.com/classgraph/classgraph/issues/282
An error occurs when calling ScanResult.close(). #285
https://github.com/classgraph/classgraph/issues/285
shutdown hook causes classloader leak when library is used inside a servlet container #376
https://github.com/classgraph/classgraph/issues/376
was (Author: klund):
Here are some ClassGraph issues (all have been fixed) which may have contributed to memory
issues involving older releases (of ClassGraph):
Memory leak of ScanResults #233
https://github.com/classgraph/classgraph/issues/233
ScanResult leaks shutdown hooks #240
https://github.com/classgraph/classgraph/issues/240
Path scanning does deep scans for whitelisted paths #282
https://github.com/classgraph/classgraph/issues/282
An error occurs when calling ScanResult.close(). #285
https://github.com/classgraph/classgraph/issues/285
shutdown hook causes classloader leak when library is used inside a servlet container #376
https://github.com/classgraph/classgraph/issues/376
> ClassGraph library causes large memory leak in Geode after adding geode-log4j
> -----------------------------------------------------------------------------
>
> Key: GEODE-7370
> URL: https://issues.apache.org/jira/browse/GEODE-7370
> Project: Geode
> Issue Type: Bug
> Components: management
> Reporter: Kirk Lund
> Assignee: Kirk Lund
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> According to the source code and documentation, io.github.classgraph.ScanResult must
be closed to avoid memory and file leaks.
> I identified a large memory leak that apparently after adding geode-log4j. Surprisingly
the cause ended up being an instance of ScanResult which is used by the management code to
search the classpath for user classes that implement geode functions or gfsh commands.
> Geode currently depends on ClassGraph version 4.0.6, but the library is up to version
4.8.52 (averaging around a dozen releases per month).
> JVM bytes in use by io.github.classgraph after closing all ScanResult instances:
> * Before geode-log4j with ClassGraph 4.0.6: 20,488 bytes
> * Before geode-log4j with ClassGraph 4.8.52: 1,056 bytes
> * After geode-log4j with ClassGraph 4.0.6: 56,753,008 bytes
> * After geode-log4j with ClassGraph 4.8.52: 1,056 bytes
> Given the above results of my testing, I believe we need to keep this dependency up-to-date
and upgrade to 4.8.52.
> ----
> More detailed notes below:
> Git version shas that were tested:
> {noformat}
> Before: 802687154131af16d350bf39d152feb4685ba7e6 (sha before geode-log4j)
> Before-w/upgrade: Before w/ ClassGraph upgraded from 4.0.6 to 4.8.52
> << Geode-log4j: efc2362d2bae0877a427ce2c29beae94118d6567 (geode-log4j) >>
> After: dffcb9446aef09c7bf6e626121f4d2ec5c74586f (async alerts)
> After-w/upgrade: After w/ ClassGraph upgraded from 4.0.6 to 4.8.52
> {noformat}
> Java process memory sizes:
> Before:
> {noformat}
> ChildVMs
> 22297 java 0.0 00:07.51 32 1 104 252M 0B 0B 22220
22286 sleeping *0[1] 0.00000 0.00000 501
> 22295 java 0.0 00:07.72 32 1 104 259M 0B 0B 22220
22286 sleeping *0[1] 0.00000 0.00000 501
> 22291 java 0.1 00:11.66 32 1 104 291M 0B 0B 22220
22286 sleeping *0[1] 0.00000 0.00000 501
> 22289 java 0.1 00:11.68 32 1 104 300M 0B 0B 22220
22286 sleeping *0[1] 0.00000 0.00000 501
> 22287 java 0.5 00:07.67 64 1 168 276M+ 0B 0B 22220
22286 sleeping *0[1] 0.00000 0.00000 501
> GradleWorkerMain
> 22286 java 0.4 00:03.55 67 1 174 184M+ 0B 0B 22220
22220 sleeping *0[1] 0.00000 0.00000 501
> GradleWrapperMain
> 22173 java 0.1 00:01.63 28 1 96 88M+ 0B 0B 22173
86991 sleeping *0[1] 0.00000 0.00000 501
> GradleDaemon
> 22220 java 0.6 00:52.27 81 1 201 814M 0B 0B 22220
22173 sleeping *0[1] 0.00000 0.00000 501
> {noformat}
> Before-w/upgrade:
> {noformat}
> ChildVMs
> 25789 java 0.0 00:06.93 31 1 101 257M 0B 0B 25714 25779
sleeping *0[1] 0.00000 0.00000 501
> 25788 java 0.1 00:06.93 31 1 101 259M+ 0B 0B 25714 25779
sleeping *0[1] 0.00000 0.00000 501
> 25784 java 0.0 00:10.37 31 1 101 286M 0B 0B 25714 25779
sleeping *0[1] 0.00000 0.00000 501
> 25782 java 0.1 00:11.68 31 1 101 278M 0B 0B 25714 25779
sleeping *0[1] 0.00000 0.00000 501
> 25780 java 0.3 00:07.84 63/1 1 165 308M+ 0B 0B 25714 25779
running *0[1] 0.00000 0.00000 501
> GradleWorkerMain
> 25779 java 0.1 00:03.39 60 1 160 183M 0B 0B 25714 25714
sleeping *0[1] 0.00000 0.00000 501
> GradleDaemon
> 25714 java 0.8 00:48.79 79 1 197 793M 0B 0B 25714 25667
sleeping *0[1] 0.00000 0.00000 501
> GradleWrapperMain
> 25667 java 0.2 00:02.16 28 1 96 101M+ 0B 0B 25667 86991
sleeping *0[1] 0.00000 0.00000 501
> {noformat}
> After:
> {noformat}
> ChildVMs
> 21056 java 0.1 00:06.67 31 1 101 256M 0B 0B 20988 21049
sleeping *0[1] 0.00000 0.00000 501
> 21055 java 0.1 00:06.82 31 1 101 264M+ 0B 0B 20988 21049
sleeping *0[1] 0.00000 0.00000 501
> 21053 java 0.1 00:15.95 31 1 101 373M 0B 0B 20988 21049
sleeping *0[1] 0.00000 0.00000 501
> 21051 java 0.3 00:14.96 31 1 101 363M+ 0B 0B 20988 21049
sleeping *0[1] 0.00000 0.00000 501
> 21050 java 0.5 00:12.07 63 1 165 343M+ 0B 0B 20988 21049
sleeping *0[1] 0.00000 0.00000 501
> GradleWorkerMain
> 21049 java 0.2 00:06.99 60 1 160 321M+ 0B 0B 20988 20988
sleeping *0[1] 0.00000 0.00000 501
> GradleWrapperMain
> 20941 java 0.2 00:02.14 28 1 96 101M+ 0B 0B 20941 86991
sleeping *0[1] 0.00000 0.00000 501
> GradleDaemon
> 20988 java 0.8 00:53.39 81 1 202 963M 0B 0B 20988 20941
sleeping *0[1] 0.00000 0.00000 501
> {noformat}
> After-w/upgrade:
> {noformat}
> ChildVMs
> 24298 java 0.1 00:06.29 32 1 104 250M 0B 0B 24231 24291
sleeping *0[1] 0.00000 0.00000 501
> 24297 java 0.1 00:06.59 32 1 104 262M 0B 0B 24231 24291
sleeping *0[1] 0.00000 0.00000 501
> 24295 java 0.1 00:14.97 32 1 104 362M 0B 0B 24231 24291
sleeping *0[1] 0.00000 0.00000 501
> 24293 java 0.2 00:15.44 32 1 104 376M+ 0B 0B 24231 24291
sleeping *0[1] 0.00000 0.00000 501
> 24292 java 0.7 00:10.36 64 1 168 327M 0B 0B 24231 24291
sleeping *0[1] 0.00000 0.00000 501
> GradleWorkerMain
> 24291 java 0.7 00:06.07 67 1 174 320M+ 0B 0B 24231 24231
sleeping *0[1] 0.00000 0.00000 501
> GradleWrapperMain
> 24184 java 0.3 00:01.39 28 1 96 89M+ 0B 0B 24184 86991
sleeping *0[1] 0.00000 0.00000 501
> GradleDaemon
> 24231 java 1.0 00:46.21 81 1 201 805M 0B 0B 24231 24184
sleeping *0[1] 0.00000 0.00000 501
> {noformat}
> Biggest objects - Dominators comparison (retained size in bytes):
> {noformat}
> Before:
> sun.misc.Launcher$AppClassLoader - 883,984
> Before-w/upgrade:
> sun.misc.Launcher$AppClassLoader - 915,384
> After:
> io.github.classgraph.ScanResult - 11,373,088
> After-w/upgrade:
> sun.misc.Launcher$AppClassLoader - 1,256,904
> {noformat}
> Class and package comparison (retained size in bytes):
> {noformat}
> org.apache.geode.internal.logging -
> Before: 253,520
> Before-w/upgrade: 253,520
> After: 33,576
> After-w/upgrade: 21,152
> org.apache.geode.logging -
> Before: 0
> Before-w/upgrade: 0
> After: 233,104
> After-w/upgrade: 231,968
> org.apache.logging.log4j -
> Before: 581,184
> Before-w/upgrade: 581,184
> After: 605,904
> After-w/upgrade: 584,576
> java.util.logging -
> Before: 45,632
> Before-w/upgrade: 45,632
> After: 45,632
> After-w/upgrade: 45,632
> {noformat}
> {noformat}
> org.apache.geode.internal.cache -
> Before: 1,034,624
> Before-w/upgrade: 1,034,560
> After: 1,065,312
> After-w/upgrade: 1,034,224
> org.apache.geode.management -
> Before: 9,592
> Before-w/upgrade: 9,592
> After: 28,408
> After-w/upgrade: 9,592
> org.apache.geode.distributed -
> Before: 224,960
> Before-w/upgrade: 224,408
> After: 256,272
> After-w/upgrade: 223,720
> {noformat}
> {noformat}
> io.github.classgraph -
> Before: 20,488
> Before-w/upgrade: 1,056
> After: 56,753,008
> After-w/upgrade: 1,056
> {noformat}
> {noformat}
> java -
> Before: 38,697,928
> Before-w/upgrade: 59,048,552
> After: 143,295,448
> After-w/upgrade: 52,397,344
> sun -
> Before: 2,464,792
> Before-w/upgrade: 2,593,712
> After: 3,359,248
> After-w/upgrade: 3,126,280
> {noformat}
> Class loader comparison:
> {noformat}
> System class loader -
> Before: 356,651 objects, 38,697,928 bytes, 4,295 classes, 3,028 classes w/o instances
> Before-w/upgrade: 883,377 objects, 59,048,552 bytes, 4,344 classes, 3,077 classes w/o
instances
> After: 1,802,231 objects, 143,295,448 bytes, 8,715 classes, 7,191 classes w/o instances
> After-w/upgrade: 712,102 objects, 52,397,344 bytes, 8,765 classes, 7,461 classes w/o
instances
> AppClassLoader -
> Before: 11,275 objects, 7,020,720 bytes, 2,885 classes, 1,896 classes w/o instances
> Before-w/upgrade: 10,467 objects, 7,068,488 bytes, 2,921 classes, 1,943 classes w/o
instances
> After: 20,336 objects, 64,013,592 bytes, 2,929 classes, 1,451 classes w/o instances
> After-w/upgrade: 10,988 objects, 7,273,232 bytes, 2,980 classes, 1,980 classes w/o
instances
> Geode ClassPathLoader$1 -
> Before: 2 objects, 130,480 bytes, 1 classes, 0 classes w/o instances
> Before-w/upgrade: 2 objects, 130,480 bytes, 1 classes, 0 classes w/o instances
> After: 2 Objects, 130,680 bytes, 1 classes, 0 classes w/o instances
> After-w/upgrade: 2 objects, 130,680 bytes, 1 classes, 0 classes w/o instances
> {noformat}
> Reachability comparison (retained size in bytes):
> {noformat}
> Objects unreachable from GC roots, but not yet collected -
> Before: 26,669,744
> Before-w/upgrade: 46,758,496
> After: 112,345,880
> After-w/upgrade: 35,093,072
> Objects reachable from GC roots via strong references -
> Before: 12,028,184
> Before-w/upgrade: 12,290,056
> After: 30,949,568
> After-w/upgrade: 17,304,272
> Objects pending finalization (finalizer queue objects unreachable via strong references)
-
> Before: 241,776
> Before-w/upgrade: 252,192
> After: 13,564,760
> After-w/upgrade: 295,160
> Objects reachable from GC roots via weak and/or soft references only -
> Before: 91,048
> Before-w/upgrade: 106,488
> After: 586,616
> After-w/upgrade: 51,616
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
|