accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-3770) Replace GarbageCollectionLogger with JVM provided GC stats
Date Wed, 14 Sep 2016 15:55:20 GMT


Josh Elser commented on ACCUMULO-3770:

Was talking with Bitweasil in IRC yesterday/today about this. He expressed some confusion
about the GC logging we had. After digging, he found that we were actually logging the cumulative
time spent in GC, not the time of the last cycle as expected.

bq.Where do these reports go?  Are they rolled-over like the regular logs?

Typically, you'll get a {{gc.log\[\-date\]}} style file, but I believe this is also configurable.

bq. The timeliness feature is nice, but that can be done in a separate SimpleTimer callback.

Agreed. Timeliness check is very good and should be considered separate to GC logs.

bq. Yeah, rollover is nice. Also does what java prints out have timestamps?

Yes. Here are some examples from the logs that HDP creates for the secondary namenode:

2016-09-14T09:57:00.476+0000: 54036.340: [GC (Allocation Failure) 2016-09-14T09:57:00.476+0000:
54036.340: [ParNew: 106050K->1130K(118016K), 0.0680896 secs] 125622K->20712K(1035520K),
0.0683270 secs] [Times: user=0.13 sys=0.00, real=0.07 secs]
2016-09-14T12:12:44.975+0000: 62180.839: [GC (Allocation Failure) 2016-09-14T12:12:44.975+0000:
62180.840: [ParNew: 106090K->1157K(118016K), 0.0503738 secs] 125672K->20742K(1035520K),
0.0505503 secs] [Times: user=0.10 sys=0.00, real=0.05 secs]
2016-09-14T14:51:46.898+0000: 71722.762: [GC (Allocation Failure) 2016-09-14T14:51:46.898+0000:
71722.762: [ParNew: 106117K->697K(118016K), 0.0705879 secs] 125702K->20662K(1035520K),
0.0707647 secs] [Times: user=0.14 sys=0.00, real=0.07 secs]

Another perk is that we can use standard tools for parsing these logs. Here is a nice little
article that explains how to read these lines

> Replace GarbageCollectionLogger with JVM provided GC stats
> ----------------------------------------------------------
>                 Key: ACCUMULO-3770
>                 URL:
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: tserver
>            Reporter: Josh Elser
>            Priority: Minor
>             Fix For: 2.0.0
> Hotspot provides some JVM opts that can be enabled which will print out status messages
for garbage collection:
> {{-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc}}
> TabletServers have the {{GarbageCollectionLogger}} doing its own inspection of heap usage
via {{Runtime}} and also tracking the real interval between calls to itself WRT the expected
interval (to try to guess if there are issues getting resources to run code).
> As long as openjdk has support for the same options, I think removing the heap inspection
would be good. Do we still want to preserve the "timeliness" check though? I'm not sure how
valuable it is in real environments.
> Thoughts?

This message was sent by Atlassian JIRA

View raw message