Return-Path: 
 <issues-return-125405-apmail-hbase-issues-archive=hbase.apache.org@hbase.apache.org>
X-Original-To: apmail-hbase-issues-archive@www.apache.org
Delivered-To: apmail-hbase-issues-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3BF091093E
	for <apmail-hbase-issues-archive@www.apache.org>;
 Wed, 27 Nov 2013 06:40:51 +0000 (UTC)
Received: (qmail 70894 invoked by uid 500); 27 Nov 2013 06:40:46 -0000
Delivered-To: apmail-hbase-issues-archive@hbase.apache.org
Received: (qmail 70777 invoked by uid 500); 27 Nov 2013 06:40:42 -0000
Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@hbase.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@hbase.apache.org>
List-Post: <mailto:issues@hbase.apache.org>
List-Id: <issues.hbase.apache.org>
Delivered-To: mailing list issues@hbase.apache.org
Received: (qmail 70474 invoked by uid 99); 27 Nov 2013 06:40:37 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Nov 2013 06:40:37 +0000
Date: Wed, 27 Nov 2013 06:40:37 +0000 (UTC)
From: "Lars Hofhansl (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12681465.1385514203121.36793.1385534437537@arcas>
In-Reply-To: <JIRA.12681465.1385514203121@arcas>
References: <JIRA.12681465.1385514203121@arcas>
Subject: [jira] [Updated] (HBASE-10046) Unmonitored HBase service could
 accumulate Status objects and OOM
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HBASE-10046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lars Hofhansl updated HBASE-10046:
----------------------------------

    Fix Version/s: 0.94.15
                   0.96.1
                   0.98.0

> Unmonitored HBase service could accumulate Status objects and OOM
> -----------------------------------------------------------------
>
>                 Key: HBASE-10046
>                 URL: https://issues.apache.org/jira/browse/HBASE-10046
>             Project: HBase
>          Issue Type: Bug
>          Components: monitoring
>    Affects Versions: 0.96.0, 0.94.14
>            Reporter: Aditya Kishore
>            Assignee: Aditya Kishore
>             Fix For: 0.98.0, 0.96.1, 0.94.15
>
>         Attachments: HBASE-10046.patch, HBASE-10046_0.94.patch
>
>
> This was observed in a cluster where HBase Master UI was not monitored for quite a while. During this period, a distributed log spitting task failed in an in-completable way and Master kept recreating the task over and over again.
> And each such task would create a monitored status...
> {code:title=SplitLogManager.java}
> .......
>   public long splitLogDistributed(final List<Path> logDirs, PathFilter filter) 
>       throws IOException {
>     MonitoredTask status = TaskMonitor.get().createStatus(
>           "Doing distributed log split in " + logDirs);
> .......
> {code}
> ...which kept accumulating on heap.
> Now these monitored tasks get cleaned only when someone looks at the service UI otherwise they keep growing boundless.
> The postmortem of the heap dump showed that these task object occupied 99% of HBase master heap accumulated over a week.



--
This message was sent by Atlassian JIRA
(v6.1#6144)