hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-19238) Port Hadoop's new GcTimeMonitor
Date Tue, 28 Nov 2017 23:39:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269721#comment-16269721
] 

Andrew Purtell commented on HBASE-19238:
----------------------------------------

Exposes a new Gauge metrics:
+  String GC_TIME_PERCENTAGE_KEY = "gcTimePercentage";
+  String GC_TIME_PRECENTAGE_DESC = "Percentage of time the JVM was paused in GC";

Configurable with these configuration parameters:
{code}
+  /** The interval over which the percentage of GC time should be calculated.
+   * A practical value would be somewhere between 30 sec and several minutes. */
+  public static final String OBSERVATION_WINDOW_MS_KEY = "gc.time.observationWindowMs";
+  public static final long DEFAULT_OBSERVATION_WINDOW_MS = 30L * 1000;
+
+  /** sleepIntervalMs how frequently this thread should wake up to check GC timings.
+   * This is also a frequency with which alertHandler will be invoked if GC time
+   * percentage exceeds the specified limit. A practical
+   * value would likely be 500..1000 ms. */
+  public static final String SLEEP_INTERVAL_MS_KEY = "gc.time.sleepIntervalMs";
+  public static final long DEFAULT_SLEEP_INTERVAL_MS = 1000;
+
+  /** maxGcTimePercentage A GC time percentage limit (0..100) within observationWindowMs.
+   * Once this is exceeded, alertHandler will be invoked every sleepIntervalMs milliseconds
+   * until GC time percentage falls below this limit. */
+  public static final String MAX_GC_TIME_PERCENTAGE_KEY = "gc.time.maxGcTimePercentage";
+  public static final int DEFAULT_MAX_GC_TIME_PERCENTAGE = 50;
{code}

As is the only thing that happens is the new metric gcTimePercentage appears in RegionServer,
REST, and Thrift server metrics sources. 

However this class can also accept an additional parameter in the constructor, a callback
to invoke in case the maxGcTimePercentage threshold is exceeded. 

{code}
+  public interface GcTimeAlertHandler {
+    void alert(GcData gcData);
+  }
{code

At some future time we might trigger an Abortable out of the GcTimeAlertHandler. If the maxGcTimePercentage
is set to a threshold like .95 because we would be in a pathological condition at that point
and the best course of action would be to terminate. (And eventually the JVM would do it for
us, but no need to wait.)

> Port Hadoop's new GcTimeMonitor
> -------------------------------
>
>                 Key: HBASE-19238
>                 URL: https://issues.apache.org/jira/browse/HBASE-19238
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Andrew Purtell
>            Assignee: Andrew Purtell
>            Priority: Minor
>         Attachments: HBASE-19238-branch-1.patch, HBASE-19238.patch
>
>
> We borrowed Hadoop's JvmPauseMonitor some time ago. On HADOOP-14960 (Add GC time percentage
monitor/alerter), Hadoop added a new GcTimeMonitor, which seems useful:
> {code}
> /**
>  * This class monitors the percentage of time the JVM is paused in GC within
>  * the specified observation window, say 1 minute. The user can provide a
>  * hook which will be called whenever this percentage exceeds the specified
>  * threshold.
>  */
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message