Return-Path: Delivered-To: apmail-hadoop-hbase-dev-archive@locus.apache.org Received: (qmail 5100 invoked from network); 17 Sep 2008 21:22:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 17 Sep 2008 21:22:05 -0000 Received: (qmail 53350 invoked by uid 500); 17 Sep 2008 21:22:02 -0000 Delivered-To: apmail-hadoop-hbase-dev-archive@hadoop.apache.org Received: (qmail 53203 invoked by uid 500); 17 Sep 2008 21:22:01 -0000 Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-dev@hadoop.apache.org Delivered-To: mailing list hbase-dev@hadoop.apache.org Received: (qmail 53192 invoked by uid 99); 17 Sep 2008 21:22:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Sep 2008 14:22:01 -0700 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Sep 2008 21:21:11 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 4A96F234C1D8 for ; Wed, 17 Sep 2008 14:21:44 -0700 (PDT) Message-ID: <1083602724.1221686504304.JavaMail.jira@brutus> Date: Wed, 17 Sep 2008 14:21:44 -0700 (PDT) From: "Jean-Daniel Cryans (JIRA)" To: hbase-dev@hadoop.apache.org Subject: [jira] Commented: (HBASE-887) Fix a hotspot in scanners In-Reply-To: <1073941233.1221679424430.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HBASE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631953#action_12631953 ] Jean-Daniel Cryans commented on HBASE-887: ------------------------------------------ Some tests I did with a modified PE. I ran the PE with a number from 10 to 3000 and incremented by 10. The garbage collector was called after each run. Here are the results: {quote} 08/09/17 16:48:54 INFO hbase.ManyPerformanceEvaluation: 10 42725 08/09/17 16:49:31 INFO hbase.ManyPerformanceEvaluation: 20 36744 08/09/17 16:50:06 INFO hbase.ManyPerformanceEvaluation: 30 34928 08/09/17 16:50:40 INFO hbase.ManyPerformanceEvaluation: 40 34150 08/09/17 16:51:14 INFO hbase.ManyPerformanceEvaluation: 50 33664 08/09/17 16:51:47 INFO hbase.ManyPerformanceEvaluation: 60 33570 08/09/17 16:52:21 INFO hbase.ManyPerformanceEvaluation: 70 33142 08/09/17 16:52:54 INFO hbase.ManyPerformanceEvaluation: 80 32901 08/09/17 16:53:26 INFO hbase.ManyPerformanceEvaluation: 90 32864 08/09/17 16:53:59 INFO hbase.ManyPerformanceEvaluation: 100 32776 08/09/17 16:54:32 INFO hbase.ManyPerformanceEvaluation: 110 32725 08/09/17 16:55:05 INFO hbase.ManyPerformanceEvaluation: 120 32747 08/09/17 16:55:38 INFO hbase.ManyPerformanceEvaluation: 130 32698 08/09/17 16:56:10 INFO hbase.ManyPerformanceEvaluation: 140 32693 08/09/17 16:56:43 INFO hbase.ManyPerformanceEvaluation: 150 32643 08/09/17 16:57:16 INFO hbase.ManyPerformanceEvaluation: 160 32686 08/09/17 16:57:48 INFO hbase.ManyPerformanceEvaluation: 170 32665 08/09/17 16:58:21 INFO hbase.ManyPerformanceEvaluation: 180 32448 08/09/17 16:58:53 INFO hbase.ManyPerformanceEvaluation: 190 32004 08/09/17 16:59:25 INFO hbase.ManyPerformanceEvaluation: 200 31985 08/09/17 16:59:57 INFO hbase.ManyPerformanceEvaluation: 210 31964 08/09/17 17:00:29 INFO hbase.ManyPerformanceEvaluation: 220 31983 08/09/17 17:01:01 INFO hbase.ManyPerformanceEvaluation: 230 31778 08/09/17 17:01:32 INFO hbase.ManyPerformanceEvaluation: 240 31766 08/09/17 17:02:04 INFO hbase.ManyPerformanceEvaluation: 250 31886 08/09/17 17:02:36 INFO hbase.ManyPerformanceEvaluation: 260 31773 08/09/17 17:03:08 INFO hbase.ManyPerformanceEvaluation: 270 31709 08/09/17 17:03:40 INFO hbase.ManyPerformanceEvaluation: 280 31669 08/09/17 17:04:11 INFO hbase.ManyPerformanceEvaluation: 290 31647 08/09/17 17:04:43 INFO hbase.ManyPerformanceEvaluation: 300 31634 08/09/17 17:05:15 INFO hbase.ManyPerformanceEvaluation: 310 31674 08/09/17 17:05:47 INFO hbase.ManyPerformanceEvaluation: 320 31935 08/09/17 17:06:18 INFO hbase.ManyPerformanceEvaluation: 330 31802 08/09/17 17:06:50 INFO hbase.ManyPerformanceEvaluation: 340 31571 08/09/17 17:07:21 INFO hbase.ManyPerformanceEvaluation: 350 31484 08/09/17 17:07:53 INFO hbase.ManyPerformanceEvaluation: 360 31713 08/09/17 17:08:25 INFO hbase.ManyPerformanceEvaluation: 370 31512 08/09/17 17:08:56 INFO hbase.ManyPerformanceEvaluation: 380 31456 08/09/17 17:09:28 INFO hbase.ManyPerformanceEvaluation: 390 32055 08/09/17 17:10:01 INFO hbase.ManyPerformanceEvaluation: 400 32260 08/09/17 17:10:33 INFO hbase.ManyPerformanceEvaluation: 410 32167 08/09/17 17:11:05 INFO hbase.ManyPerformanceEvaluation: 420 31962 08/09/17 17:11:37 INFO hbase.ManyPerformanceEvaluation: 430 32003 08/09/17 17:12:09 INFO hbase.ManyPerformanceEvaluation: 440 32054 08/09/17 17:12:41 INFO hbase.ManyPerformanceEvaluation: 450 32105 08/09/17 17:13:13 INFO hbase.ManyPerformanceEvaluation: 460 32126 08/09/17 17:13:45 INFO hbase.ManyPerformanceEvaluation: 470 31961 08/09/17 17:14:17 INFO hbase.ManyPerformanceEvaluation: 480 31968 ... {quote} It seems that 2000 was an overkill. The gain is impressive between 1 and 10 then around 300 we pretty much reduced the RPC call at it's smallest value. > Fix a hotspot in scanners > ------------------------- > > Key: HBASE-887 > URL: https://issues.apache.org/jira/browse/HBASE-887 > Project: Hadoop HBase > Issue Type: Improvement > Components: client, regionserver > Reporter: Jean-Daniel Cryans > Assignee: Jean-Daniel Cryans > Fix For: 0.19.0 > > Attachments: hbase-887-v1.patch > > > When scanning, we do a lot of RPCs and this has a huge performance hit. I propose that we add a way to fetch more rows during next() and put them in cache. This should be configurable. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.