Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 29 Mar 2017 16:45:41 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13027152.1481330894000.156106.1490805941990@Atlassian.JIRA>
In-Reply-To: <JIRA.13027152.1481330894000@Atlassian.JIRA>
References: <JIRA.13027152.1481330894000@Atlassian.JIRA> <JIRA.13027152.1481330894163@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-17287) Master becomes a zombie if
 filesystem object closes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 29 Mar 2017 16:45:52 -0000


    [ https://issues.apache.org/jira/browse/HBASE-17287?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D159=
47482#comment-15947482 ]=20

Hudson commented on HBASE-17287:
--------------------------------

SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #115 (See [https://buil=
ds.apache.org/job/HBase-1.2-JDK8/115/])
HBASE-17287 Master becomes a zombie if filesystem object closes (tedyu: rev=
 2d79b7d5a508c2175312487db3e93d838e063ec2)
* (add) hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure=
/TestSafemodeBringsDownMaster.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFi=
leSystem.java


> Master becomes a zombie if filesystem object closes
> ---------------------------------------------------
>
>                 Key: HBASE-17287
>                 URL: https://issues.apache.org/jira/browse/HBASE-17287
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>            Reporter: Clay B.
>            Assignee: Ted Yu
>            Priority: Blocker
>             Fix For: 1.4.0, 1.3.1, 1.1.9, 2.0, 1.2.6
>
>         Attachments: 17287.branch-1.1.v4.txt, 17287.branch-1.v3.txt, 1728=
7.branch-1.v4.txt, 17287.master.v2.txt, 17287.master.v3.txt, 17287.master.v=
4.txt, 17287.master.v5.txt, 17287.v2.txt
>
>
> We have seen an issue whereby if the HDFS is unstable and the HBase maste=
r's HDFS client is unable to stabilize before {{dfs.client.failover.max.att=
empts}} then the master's filesystem object closes. This seems to result in=
 an HBase master which will continue to run (process and znode exists) but =
no meaningful work can be done (e.g. assigning meta).What we saw in our HBa=
se master logs was:{code}2016-12-01 19:19:08,192 ERROR org.apache.hadoop.hb=
ase.master.handler.MetaServerShutdownHandler: Caught M_META_SERVER_SHUTDOWN=
, count=3D1java.io.IOException: failed log splitting for cluster-r5n12.bloo=
mberg.com,60200,1480632863218, will retryat org.apache.hadoop.hbase.master.=
handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:84=
)at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129=
)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja=
va:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe=
cutor.java:617)at java.lang.Thread.run(Thread.java:745)Caused by: java.io.I=
OException: Filesystem closed{code}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)