Return-Path: X-Original-To: apmail-ambari-issues-archive@minotaur.apache.org Delivered-To: apmail-ambari-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 81ABF19248 for ; Wed, 9 Mar 2016 21:40:46 +0000 (UTC) Received: (qmail 34580 invoked by uid 500); 9 Mar 2016 21:40:41 -0000 Delivered-To: apmail-ambari-issues-archive@ambari.apache.org Received: (qmail 34495 invoked by uid 500); 9 Mar 2016 21:40:41 -0000 Mailing-List: contact issues-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list issues@ambari.apache.org Received: (qmail 34453 invoked by uid 99); 9 Mar 2016 21:40:41 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2016 21:40:41 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id C54CB2C1F55 for ; Wed, 9 Mar 2016 21:40:40 +0000 (UTC) Date: Wed, 9 Mar 2016 21:40:40 +0000 (UTC) From: "Hudson (JIRA)" To: issues@ambari.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (AMBARI-15355) After EU from 2.2 -> 2.3 HBase region server started failed with memory error MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AMBARI-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188088#comment-15188088 ] Hudson commented on AMBARI-15355: --------------------------------- FAILURE: Integrated in Ambari-branch-2.2 #491 (See [https://builds.apache.org/job/Ambari-branch-2.2/491/]) AMBARI-15355. After EU from 2.2 -> 2.3 HBase region server started (dlysnichenko: [http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=4051f690b919c8940c7e66bed4e59dc7854039ab]) * ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py * ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > After EU from 2.2 -> 2.3 HBase region server started failed with memory error > ----------------------------------------------------------------------------- > > Key: AMBARI-15355 > URL: https://issues.apache.org/jira/browse/AMBARI-15355 > Project: Ambari > Issue Type: Bug > Components: ambari-server > Reporter: Dmitry Lysnichenko > Assignee: Dmitry Lysnichenko > Attachments: AMBARI-15355.patch > > > On one of the clusters i did EU from 2.2.x to 2.3.x. > During upgrade there were problems with HBase service checks for region servers and thus upgrade is paused. > Region server start is failing with error > {code} > 2016-03-03 19:55:31,203 ERROR [regionserver:16020] regionserver.HRegionServer: Failed init > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > at org.apache.hadoop.hbase.util.ByteBufferArray.(ByteBufferArray.java:65) > at org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.(ByteBufferIOEngine.java:47) > at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307) > at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:217) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.(CacheConfig.java:231) > at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361) > at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899) > at java.lang.Thread.run(Thread.java:745) > 2016-03-03 19:55:31,206 FATAL [regionserver:16020] regionserver.RSRpcServices: Run out of memory; RSRpcServices will abort itself immediately > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > at org.apache.hadoop.hbase.util.ByteBufferArray.(ByteBufferArray.java:65) > at org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.(ByteBufferIOEngine.java:47) > at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307) > at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:217) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.(CacheConfig.java:231) > at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361) > at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899) > at java.lang.Thread.run(Thread.java:745) > 2016-03-03 19:55:35,138 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-3485--1, built on 12/16/2015 02:35 GMT > {code} > This was seen on the following cluster: https://s.c:8443/#/main/services/HBASE/configs -- This message was sent by Atlassian JIRA (v6.3.4#6332)