Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4EE2A10ED1 for ; Thu, 3 Sep 2015 10:29:46 +0000 (UTC) Received: (qmail 51074 invoked by uid 500); 3 Sep 2015 10:29:46 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 51025 invoked by uid 500); 3 Sep 2015 10:29:46 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 51014 invoked by uid 99); 3 Sep 2015 10:29:46 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Sep 2015 10:29:46 +0000 Date: Thu, 3 Sep 2015 10:29:46 +0000 (UTC) From: "Mukund Murrali (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-13942) HBase client stalls during region split when client threads > hbase.hconnection.threads.max MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-13942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728830#comment-14728830 ] Mukund Murrali commented on HBASE-13942: ---------------------------------------- It is becoming increasingly difficult with having 256 threads/cluster. Is it not possible to reduce this? We are afraid the same issue might prop up in case of drastically reducing the threads. > HBase client stalls during region split when client threads > hbase.hconnection.threads.max > ------------------------------------------------------------------------------------------- > > Key: HBASE-13942 > URL: https://issues.apache.org/jira/browse/HBASE-13942 > Project: HBase > Issue Type: Bug > Components: Client, regionserver > Reporter: Mukund Murrali > > Performing any operataion using a single hconnection with client threads > hbase.hconnection.threads.max causing the client to stall indefinetly during first region split. All the hconnection threads in client side are waiting with the following stack. > hconnection-0x648a83fd-shared--pool1-t8" daemon prio=10 tid=0x00007f447c003800 nid=0x62ff waiting on condition [0x00007f44c72f0000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007d768bdf0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374) > at org.apache.hadoop.hbase.util.BoundedCompletionService.take(BoundedCompletionService.java:74) > at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:174) > at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:56) > at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) > at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:145) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1200) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1109) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncProcess.java:916) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:833) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1156) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1296) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:574) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:716) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > -- This message was sent by Atlassian JIRA (v6.3.4#6332)