Return-Path: <issues-return-250825-archive-asf-public=cust-asf.ponee.io@hbase.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id 0A2E5200B4F
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Mon, 11 Jul 2016 12:22:13 +0200 (CEST)
Received: by cust-asf.ponee.io (Postfix)
	id 09006160A78; Mon, 11 Jul 2016 10:22:13 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id 5518D160A62
	for <archive-asf-public@cust-asf.ponee.io>; Mon, 11 Jul 2016 12:22:12 +0200 (CEST)
Received: (qmail 46066 invoked by uid 500); 11 Jul 2016 10:22:11 -0000
Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@hbase.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@hbase.apache.org>
List-Post: <mailto:issues@hbase.apache.org>
List-Id: <issues.hbase.apache.org>
Delivered-To: mailing list issues@hbase.apache.org
Received: (qmail 46027 invoked by uid 99); 11 Jul 2016 10:22:11 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jul 2016 10:22:11 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id 225902C02A7
	for <issues@hbase.apache.org>; Mon, 11 Jul 2016 10:22:11 +0000 (UTC)
Date: Mon, 11 Jul 2016 10:22:11 +0000 (UTC)
From: "Konstantin Ryakhovskiy (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12984774.1467178103000.75626.1468232531137@Atlassian.JIRA>
In-Reply-To: <JIRA.12984774.1467178103000@Atlassian.JIRA>
References: <JIRA.12984774.1467178103000@Atlassian.JIRA> <JIRA.12984774.1467178103983@arcas>
Subject: [jira] [Updated] (HBASE-16142) Trigger JFR session when under
 duress -- e.g. backed-up request queue count -- and dump the recording to
 log dir
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
archived-at: Mon, 11 Jul 2016 10:22:13 -0000


     [ https://issues.apache.org/jira/browse/HBASE-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Ryakhovskiy updated HBASE-16142:
-------------------------------------------
    Attachment: HBASE-16142.master.001.patch

Added class JavaFlightRecorder and test.
The class has a main method to be called with command line arguments:
--duration=<duration> [--output=<path>] 
if output is not provided, JFR trace will be dumped into HBase log directory.
the JVM option UnlockCommercialFeatures has to be enabled.

Do not know how to test if UnlockCommercialFeatures is disabled/enabled by default. Currently, tests are just skipped if the option is disabled.

The patch on the review board:
https://reviews.apache.org/r/49888/


 

> Trigger JFR session when under duress -- e.g. backed-up request queue count -- and dump the recording to log dir
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-16142
>                 URL: https://issues.apache.org/jira/browse/HBASE-16142
>             Project: HBase
>          Issue Type: Task
>          Components: Operability
>            Reporter: stack
>            Assignee: Konstantin Ryakhovskiy
>            Priority: Minor
>              Labels: beginner
>         Attachments: HBASE-16142.master.001.patch
>
>
> Chatting today w/ a mighty hbase operator on how to figure what is happening during transitory latency spike or any other transitory 'weirdness' in a server, the idea came up that a java flight recording during a spike would include a pretty good picture of what is going on during the time of duress (more ideal would be a trace of the explicit slow queries showing call stack with timings dumped to a sink for later review; i.e. trigger an htrace when a query is slow...).
> Taking a look, programmatically triggering a JFR recording seems doable, if awkward (MBean invocations). There is even a means of specifying 'triggers' based off any published mbean emission -- e.g. a query queue count threshold -- which looks nice. See https://community.oracle.com/thread/3676275?start=0&tstart=0 and https://docs.oracle.com/javacomponents/jmc-5-4/jfr-runtime-guide/run.htm#JFRUH184
> This feature could start out as a blog post describing how to do it for one server. A plugin on Canary that looks at mbean values and if over a configured threshold, triggers a recording remotely could be next. Finally could integrate a couple of triggers that fire when issue via the trigger mechanism.
> Marking as beginner feature.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)