Return-Path: 
 <issues-return-7505-apmail-ambari-issues-archive=ambari.apache.org@ambari.apache.org>
X-Original-To: apmail-ambari-issues-archive@minotaur.apache.org
Delivered-To: apmail-ambari-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 86E97195AA
	for <apmail-ambari-issues-archive@minotaur.apache.org>;
 Tue, 26 Apr 2016 13:52:13 +0000 (UTC)
Received: (qmail 90184 invoked by uid 500); 26 Apr 2016 13:52:13 -0000
Delivered-To: apmail-ambari-issues-archive@ambari.apache.org
Received: (qmail 90115 invoked by uid 500); 26 Apr 2016 13:52:13 -0000
Mailing-List: contact issues-help@ambari.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@ambari.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@ambari.apache.org>
List-Post: <mailto:issues@ambari.apache.org>
List-Id: <issues.ambari.apache.org>
Reply-To: dev@ambari.apache.org
Delivered-To: mailing list issues@ambari.apache.org
Received: (qmail 89971 invoked by uid 99); 26 Apr 2016 13:52:13 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Apr 2016 13:52:13 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id 38A2F2C1F62
	for <issues@ambari.apache.org>; Tue, 26 Apr 2016 13:52:13 +0000 (UTC)
Date: Tue, 26 Apr 2016 13:52:13 +0000 (UTC)
From: "Dmitry Lysnichenko (JIRA)" <jira@apache.org>
To: issues@ambari.apache.org
Message-ID: <JIRA.12962596.1461678710000.34775.1461678733229@Atlassian.JIRA>
In-Reply-To: <JIRA.12962596.1461678710000@Atlassian.JIRA>
References: <JIRA.12962596.1461678710000@Atlassian.JIRA>
 <JIRA.12962596.1461678710957@arcas>
Subject: [jira] [Created] (AMBARI-16121) HBase RegionServers go down after
 Ambari upgrade due to ClassCastException
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Dmitry Lysnichenko created AMBARI-16121:
-------------------------------------------

             Summary: HBase RegionServers go down after Ambari upgrade due to ClassCastException
                 Key: AMBARI-16121
                 URL: https://issues.apache.org/jira/browse/AMBARI-16121
             Project: Ambari
          Issue Type: Bug
            Reporter: Dmitry Lysnichenko
            Assignee: Dmitry Lysnichenko
         Attachments: AMBARI-16121.patch



ambari-server --hash
2b112376b631384852a6c8aaa2e102d8dd39a9f1
ambari-server-2.2.2.0-456.x86_64

*Steps*
# Deploy HDP 2.3.4.0 cluster with Ambari 2.2.0.0 (HA, unsecure cluster and Ranger enabled)
# Upgrade Ambari to 2.2.2.0

Observed that all HBase RS reported as down

Logs show below error:
{code}
016-04-26 09:47:23,575 ERROR [regionserver/host1/ip:16020] coprocessor.CoprocessorHost: The coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw java.lang.ClassCastException: org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$RegionServerEnvironment cannot be cast to org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment
java.lang.ClassCastException: org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$RegionServerEnvironment cannot be cast to org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment
at org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.start(SecureBulkLoadEndpoint.java:131)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:411)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:253)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:156)
at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.<init>(RegionServerCoprocessorHost.java:69)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:888)
at java.lang.Thread.run(Thread.java:745)
2016-04-26 09:47:23,584 FATAL [regionserver/host1/ip:16020] regionserver.HRegionServer: ABORTING region server host1,16020,1461664036377: The coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw java.lang.ClassCastException: org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$RegionServerEnvironment cannot be cast to org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment
java.lang.ClassCastException: org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$RegionServerEnvironment cannot be cast to org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment
at org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.start(SecureBulkLoadEndpoint.java:131)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:411)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:253)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:156)
at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.<init>(RegionServerCoprocessorHost.java:69)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:888)
at java.lang.Thread.run(Thread.java:745)
2016-04-26 09:47:23,586 FATAL [regionserver/host1/ip:16020] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.token.TokenProvider]
2016-04-26 09:47:27,474 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2651)
2016-04-26 09:47:27,491 INFO  [Thread-129] provider.AuditProviderFactory: ==> JVMShutdownHook.run()
2016-04-26 09:47:27,492 INFO  [Thread-129] queue.AuditAsyncQueue: Stop called. name=hbaseRegional.async
2016-04-26 09:47:27,492 INFO  [Thread-129] queue.AuditAsyncQueue: Interrupting consumerThread. name=hbaseRegional.async, consumer=hbaseRegional.async.summary
2016-04-26 09:47:27,496 INFO  [Thread-129] provider.AuditProviderFactory: <== JVMShutdownHook.run()
2016-04-26 09:47:27,507 INFO  [Thread-7] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@16423501
2016-04-26 09:47:27,506 INFO  [org.apache.ranger.audit.queue.AuditAsyncQueue0] queue.AuditAsyncQueue: Caught exception in consumer thread. Shutdown might be in progress
{code}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)