Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AC34318DBD for ; Tue, 25 Aug 2015 22:30:47 +0000 (UTC) Received: (qmail 833 invoked by uid 500); 25 Aug 2015 22:30:47 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 793 invoked by uid 500); 25 Aug 2015 22:30:47 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 774 invoked by uid 99); 25 Aug 2015 22:30:47 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2015 22:30:47 +0000 Date: Tue, 25 Aug 2015 22:30:47 +0000 (UTC) From: "Hudson (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-10785) Metas own location should be cached MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-10785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712087#comment-14712087 ] Hudson commented on HBASE-10785: -------------------------------- SUCCESS: Integrated in HBase-0.98 #1099 (See [https://builds.apache.org/job/HBase-0.98/1099/]) HBASE-14275 Backport to 0.98 HBASE-10785 Metas own location should be cached (jerryjch: rev c0abda2e110772c58b5520789ffd53f287b399cf) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java > Metas own location should be cached > ----------------------------------- > > Key: HBASE-10785 > URL: https://issues.apache.org/jira/browse/HBASE-10785 > Project: HBase > Issue Type: Improvement > Reporter: Enis Soztutar > Assignee: Enis Soztutar > Fix For: 0.99.0, hbase-10070 > > Attachments: 0034-HBASE-10785-Metas-own-location-should-be-cached.patch, hbase-10785_v1.patch, hbase-10785_v2.patch, hbase-10785_v3.patch > > > With ROOT table gone, we no longer cache the location of the meta table (in MetaCache) in 96+. I've checked 94 code, and there we cache meta, but not root. > However, not caching the metas own location means that we are doing a zookeeper request every time we want to look up a regions location from meta. This means that there is a significant spike in zk requests whenever a region server goes down. > This affects trunk,0.98 and 0.96 as well as hbase-10070 branch. I've discovered the issue in hbase-10070 because of the integration test (HBASE-10572) results in 150K requests to zk in 10min. > A thread dump from one of the runs have 100+ threads from client in this stack trace: > {code} > "TimeBoundedMultiThreadedReaderThread_20" prio=10 tid=0x00007f852c2f2000 nid=0x57b6 in Object.wait() [0x00007f85059e7000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:503) > at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309) > - locked <0x00000000ea71aa78> (a org.apache.zookeeper.ClientCnxn$Packet) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149) > at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:337) > at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:684) > at org.apache.hadoop.hbase.zookeeper.ZKUtil.blockUntilAvailable(ZKUtil.java:1853) > at org.apache.hadoop.hbase.zookeeper.MetaRegionTracker.blockUntilAvailable(MetaRegionTracker.java:186) > at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1126) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1112) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1220) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1129) > at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:321) > at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:257) > - locked <0x00000000e9bcf238> (a org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:818) > at org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.queryKey(MultiThreadedReader.java:288) > at org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.readKey(MultiThreadedReader.java:249) > at org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.runReader(MultiThreadedReader.java:192) > at org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.run(MultiThreadedReader.java:150) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)