hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-10046) Unmonitored HBase service could accumulate Status objects and OOM
Date Wed, 27 Nov 2013 16:24:40 GMT

     [ https://issues.apache.org/jira/browse/HBASE-10046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ted Yu updated HBASE-10046:
---------------------------

    Fix Version/s:     (was: 0.96.1)
                   0.96.2

> Unmonitored HBase service could accumulate Status objects and OOM
> -----------------------------------------------------------------
>
>                 Key: HBASE-10046
>                 URL: https://issues.apache.org/jira/browse/HBASE-10046
>             Project: HBase
>          Issue Type: Bug
>          Components: monitoring
>    Affects Versions: 0.96.0, 0.94.14
>            Reporter: Aditya Kishore
>            Assignee: Aditya Kishore
>             Fix For: 0.98.0, 0.94.15, 0.96.2
>
>         Attachments: HBASE-10046.patch, HBASE-10046_0.94.patch
>
>
> This was observed in a cluster where HBase Master UI was not monitored for quite a while.
During this period, a distributed log spitting task failed in an in-completable way and Master
kept recreating the task over and over again.
> And each such task would create a monitored status...
> {code:title=SplitLogManager.java}
> .......
>   public long splitLogDistributed(final List<Path> logDirs, PathFilter filter)

>       throws IOException {
>     MonitoredTask status = TaskMonitor.get().createStatus(
>           "Doing distributed log split in " + logDirs);
> .......
> {code}
> ...which kept accumulating on heap.
> Now these monitored tasks get cleaned only when someone looks at the service UI otherwise
they keep growing boundless.
> The postmortem of the heap dump showed that these task object occupied 99% of HBase master
heap accumulated over a week.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message