Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 39E9011F87 for ; Sat, 28 Jun 2014 06:14:38 +0000 (UTC) Received: (qmail 57227 invoked by uid 500); 28 Jun 2014 06:14:35 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 57196 invoked by uid 500); 28 Jun 2014 06:14:35 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 57162 invoked by uid 99); 28 Jun 2014 06:14:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Jun 2014 06:14:34 +0000 Date: Sat, 28 Jun 2014 06:14:34 +0000 (UTC) From: "Hudson (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-10785) Metas own location should be cached MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-10785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046759#comment-14046759 ] Hudson commented on HBASE-10785: -------------------------------- FAILURE: Integrated in HBase-TRUNK #5245 (See [https://builds.apache.org/job/HBase-TRUNK/5245/]) HBASE-10785 Metas own location should be cached (enis: rev 48ffa4d5e615c78f6db8f6c2beddd93460887642) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java HBASE-11332 Fix for metas location cache from HBASE-10785 (enis: rev 14a09e79bdb9b925ce9548b45fc4e26caadaec07) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java > Metas own location should be cached > ----------------------------------- > > Key: HBASE-10785 > URL: https://issues.apache.org/jira/browse/HBASE-10785 > Project: HBase > Issue Type: Improvement > Reporter: Enis Soztutar > Assignee: Enis Soztutar > Fix For: 0.99.0, hbase-10070 > > Attachments: 0034-HBASE-10785-Metas-own-location-should-be-cached.patch, hbase-10785_v1.patch, hbase-10785_v2.patch, hbase-10785_v3.patch > > > With ROOT table gone, we no longer cache the location of the meta table (in MetaCache) in 96+. I've checked 94 code, and there we cache meta, but not root. > However, not caching the metas own location means that we are doing a zookeeper request every time we want to look up a regions location from meta. This means that there is a significant spike in zk requests whenever a region server goes down. > This affects trunk,0.98 and 0.96 as well as hbase-10070 branch. I've discovered the issue in hbase-10070 because of the integration test (HBASE-10572) results in 150K requests to zk in 10min. > A thread dump from one of the runs have 100+ threads from client in this stack trace: > {code} > "TimeBoundedMultiThreadedReaderThread_20" prio=10 tid=0x00007f852c2f2000 nid=0x57b6 in Object.wait() [0x00007f85059e7000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:503) > at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309) > - locked <0x00000000ea71aa78> (a org.apache.zookeeper.ClientCnxn$Packet) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149) > at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:337) > at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:684) > at org.apache.hadoop.hbase.zookeeper.ZKUtil.blockUntilAvailable(ZKUtil.java:1853) > at org.apache.hadoop.hbase.zookeeper.MetaRegionTracker.blockUntilAvailable(MetaRegionTracker.java:186) > at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1126) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1112) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1220) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1129) > at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:321) > at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:257) > - locked <0x00000000e9bcf238> (a org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:818) > at org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.queryKey(MultiThreadedReader.java:288) > at org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.readKey(MultiThreadedReader.java:249) > at org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.runReader(MultiThreadedReader.java:192) > at org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.run(MultiThreadedReader.java:150) > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)