Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 6028B200B66 for ; Wed, 3 Aug 2016 23:42:22 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 5ECA3160A86; Wed, 3 Aug 2016 21:42:22 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7EBC2160A5D for ; Wed, 3 Aug 2016 23:42:21 +0200 (CEST) Received: (qmail 71988 invoked by uid 500); 3 Aug 2016 21:42:20 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 71971 invoked by uid 99); 3 Aug 2016 21:42:20 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Aug 2016 21:42:20 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 868A82C027F for ; Wed, 3 Aug 2016 21:42:20 +0000 (UTC) Date: Wed, 3 Aug 2016 21:42:20 +0000 (UTC) From: "Enis Soztutar (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HBASE-16350) Undo server abort from HBASE-14968 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Wed, 03 Aug 2016 21:42:22 -0000 [ https://issues.apache.org/jira/browse/HBASE-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-16350: ---------------------------------- Attachment: hbase-16350_v1.patch We can abort in case of Error or RuntimeException. > Undo server abort from HBASE-14968 > ---------------------------------- > > Key: HBASE-16350 > URL: https://issues.apache.org/jira/browse/HBASE-16350 > Project: HBase > Issue Type: Bug > Reporter: Enis Soztutar > Assignee: Enis Soztutar > Attachments: hbase-16350_v1.patch > > > This came up during testing the ITBLL on a custom cluster. > Master aborts if WAL splitting fails: > {code} > 2016-08-03 19:56:57,693 INFO [B.priority.fifo.QRpcServer.handler=4,queue=0,port=20000] master.ServerManager: Registering server=hbase-bug58335-6.openstacklocal,16020,1470254212713 > 2016-08-03 19:57:00,062 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase-secure/splitWAL/WALs%2Fhbase-bug58335-6.openstacklocal%2C16020%2C1470253655579-splitting%2Fhbase-bug58335-6.openstacklocal%252C16020%252C1470253655579.default.1470253660327 entered state: ERR hbase-bug58335-4.openstacklocal,16020,1470253592920 > 2016-08-03 19:57:00,064 WARN [main-EventThread] coordination.SplitLogManagerCoordination: Error splitting /hbase-secure/splitWAL/WALs%2Fhbase-bug58335-6.openstacklocal%2C16020%2C1470253655579-splitting%2Fhbase-bug58335-6.openstacklocal%252C16020%252C1470253655579.default.1470253660327 > 2016-08-03 19:57:00,064 WARN [MASTER_SERVER_OPERATIONS-hbase-bug58335-7:20000-0] master.SplitLogManager: error while splitting logs in [hdfs://hbase-bug58335-7.openstacklocal:8020/apps/hbase/data/WALs/hbase-bug58335-6.openstacklocal,16020,1470253655579-splitting] installed = 1 but only 0 done > 2016-08-03 19:57:00,065 ERROR [MASTER_SERVER_OPERATIONS-hbase-bug58335-7:20000-0] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN > java.io.IOException: failed log splitting for hbase-bug58335-6.openstacklocal,16020,1470253655579, will retry > at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:357) > at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:220) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: error or interrupted while splitting logs in [hdfs://hbase-bug58335-7.openstacklocal:8020/apps/hbase/data/WALs/hbase-bug58335-6.openstacklocal,16020,1470253655579-splitting] Task = installed = 1 done = 0 error = 1 > at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:290) > at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:393) > at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:366) > at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288) > at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:213) > ... 4 more > 2016-08-03 19:57:00,067 FATAL [MASTER_SERVER_OPERATIONS-hbase-bug58335-7:20000-0] master.HMaster: Master server abort: loaded coprocessors are: [org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor, org.apache.hadoop.hbase.backup.master.BackupController] > 2016-08-03 19:57:00,067 FATAL [MASTER_SERVER_OPERATIONS-hbase-bug58335-7:20000-0] master.HMaster: Caught throwable while processing event M_SERVER_SHUTDOWN > java.io.IOException: failed log splitting for hbase-bug58335-6.openstacklocal,16020,1470253655579, will retry > at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:357) > at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:220) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: error or interrupted while splitting logs in [hdfs://hbase-bug58335-7.openstacklocal:8020/apps/hbase/data/WALs/hbase-bug58335-6.openstacklocal,16020,1470253655579-splitting] Task = installed = 1 done = 0 error = 1 > at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:290) > at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:393) > at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:366) > at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288) > at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:213) > ... 4 more > {code} > If we fail to split the WAL, we normally retry, but it seems that it is doing this by throwing an exception: > {code} > private void resubmit(final ServerName serverName, IOException ex) throws IOException { > // typecast to SSH so that we make sure that it is the SSH instance that > // gets submitted as opposed to MSSH or some other derived instance of SSH > this.services.getExecutorService().submit((ServerShutdownHandler) this); > this.deadServers.add(serverName); > throw new IOException("failed log splitting for " + serverName + ", will retry", ex); > } > {code} > In erie, this patch: HBASE-14968 made the change that if the handler is throwing an uncaught exception, we abort the server: > {code} > protected void handleException(Throwable t) { > String msg = "Caught throwable while processing event " + eventType; > LOG.error(msg, t); > if (server != null) { > server.abort(msg, t); > } > } > {code} > It seems that SSH, and some other handlers are throwing exceptions in valid cases to retry the operations. We can undo the server.abort() call which was originally added in HBASE-14968 for cases where unchecked exceptions like ConcurrentModificationException, etc are not ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)