Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3595B103EB for ; Fri, 19 Apr 2013 09:57:39 +0000 (UTC) Received: (qmail 67156 invoked by uid 500); 19 Apr 2013 09:57:37 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 66978 invoked by uid 500); 19 Apr 2013 09:57:36 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 66965 invoked by uid 99); 19 Apr 2013 09:57:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Apr 2013 09:57:36 +0000 X-ASF-Spam-Status: No, hits=0.6 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS,UNPARSEABLE_RELAY,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [216.82.254.109] (HELO mail1.bemta7.messagelabs.com) (216.82.254.109) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Apr 2013 09:57:31 +0000 Received: from [216.82.254.19:2885] by server-13.bemta-7.messagelabs.com id F3/23-22723-6F411715; Fri, 19 Apr 2013 09:57:10 +0000 X-Env-Sender: Omkar.Joshi@lntinfotech.com X-Msg-Ref: server-2.tower-96.messagelabs.com!1366365426!1273101!1 X-Originating-IP: [203.199.118.205] X-StarScan-Received: X-StarScan-Version: 6.8.6.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 23468 invoked from network); 19 Apr 2013 09:57:09 -0000 Received: from unknown (HELO VSHINMSHTCAS01.vshodc.lntinfotech.com) (203.199.118.205) by server-2.tower-96.messagelabs.com with AES128-SHA encrypted SMTP; 19 Apr 2013 09:57:09 -0000 Received: from vshinmsmbx01.vshodc.lntinfotech.com ([172.17.24.118]) by VSHINMSHTCAS01.vshodc.lntinfotech.com ([172.17.24.112]) with mapi; Fri, 19 Apr 2013 15:25:34 +0530 From: Omkar Joshi To: "user@hbase.apache.org" Date: Fri, 19 Apr 2013 15:25:31 +0530 Subject: RE: Speeding up the row count Thread-Topic: Speeding up the row count Thread-Index: Ac484hCRYlAgxCWpQP2trGU6KTxwEwAAEmlg Message-ID: References: <1366224730346-4042415.post@n3.nabble.com> <4A68BC67-9820-4E31-844B-2D4F9260E74B@gmail.com> In-Reply-To: <4A68BC67-9820-4E31-844B-2D4F9260E74B@gmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-cr-puzzleid: {FA2282FD-2FC7-4138-AF9A-5C6C8A4CF4C0} x-cr-hashedpuzzle: zog= AAWM AkIv Ao0E A+wo CneF DGtk DaPu EUGy F0MN GXox H6P8 JP6L JlWj KO7K KTOR;1;dQBzAGUAcgBAAGgAYgBhAHMAZQAuAGEAcABhAGMAaABlAC4AbwByAGcA;Sosha1_v1;7;{FA2282FD-2FC7-4138-AF9A-5C6C8A4CF4C0};bwBtAGsAYQByAC4AagBvAHMAaABpAEAAbABuAHQAaQBuAGYAbwB0AGUAYwBoAC4AYwBvAG0A;Fri, 19 Apr 2013 09:55:31 GMT;UgBFADoAIABTAHAAZQBlAGQAaQBuAGcAIAB1AHAAIAB0AGgAZQAgAHIAbwB3ACAAYwBvAHUAbgB0AA== acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Hi Ted, 6 minutes is too long :( Will this decrease to seconds if more nodes are added in the cluster? I got this exception finally(I recall faintly about increasing some timeout= parameter while querying but I didn't want to increase it to a high value)= : Apr 19, 2013 1:05:43 PM org.apache.hadoop.hbase.client.HConnectionManager$H= ConnectionImplementation processExecs WARNING: Error executing for row java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.client.Ret= riesExhaustedException: Failed after attempts=3D10, exceptions: Fri Apr 19 12:56:01 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1770 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 12:57:02 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1782 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 12:58:04 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1785 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 12:59:05 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1794 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:00:08 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1800 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:01:10 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1802 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:02:14 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1804 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:03:19 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1809 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:04:27 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1812 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:05:43 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1829 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) at java.util.concurrent.FutureTask.get(Unknown Source) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImp= lementation.processExecs(HConnectionManager.java:1475) at org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.jav= a:1236) at org.apache.hadoop.hbase.client.coprocessor.AggregationClient.row= Count(AggregationClient.java:216) at client.hbase.HBaseCRUD.getTableCount(HBaseCRUD.java:307) at client.hbase.HBaseCRUD.main(HBaseCRUD.java:117) Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed= after attempts=3D10, exceptions: Fri Apr 19 12:56:01 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1770 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 12:57:02 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1782 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 12:58:04 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1785 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 12:59:05 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1794 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:00:08 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1800 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:01:10 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1802 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:02:14 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1804 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:03:19 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1809 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:04:27 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1812 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:05:43 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1829 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] at org.apache.hadoop.hbase.client.ServerCallable.withRetries(Server= Callable.java:183) at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker= .java:79) at $Proxy6.getRowNum(Unknown Source) at org.apache.hadoop.hbase.client.coprocessor.AggregationClient$3.c= all(AggregationClient.java:220) at org.apache.hadoop.hbase.client.coprocessor.AggregationClient$3.c= all(AggregationClient.java:217) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImp= lementation$4.call(HConnectionManager.java:1463) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source= ) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Sourc= e) at java.lang.Thread.run(Unknown Source) Apr 19, 2013 1:05:43 PM org.apache.hadoop.hbase.client.HConnectionManager$H= ConnectionImplementation internalClose INFO: Closed zookeeper sessionid=3D0x13e185b8ee8003a org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after atte= mpts=3D10, exceptions: Fri Apr 19 12:56:01 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1770 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 12:57:02 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1782 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 12:58:04 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1785 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 12:59:05 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1794 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:00:08 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1800 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:01:10 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1802 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:02:14 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1804 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:03:19 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1809 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:04:27 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1812 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] Fri Apr 19 13:05:43 IST 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@= 1d6e77, java.net.SocketTimeoutException: Call to cldx-1140-1034/172.25.6.71= :60020 failed on socket timeout exception: java.net.SocketTimeoutException:= 60000 millis timeout while waiting for channel to be ready for read. ch : = java.nio.channels.SocketChannel[connected local=3D/0.0.0.0:1829 remote=3Dcl= dx-1140-1034/172.25.6.71:60020] at org.apache.hadoop.hbase.client.ServerCallable.withRetries(Server= Callable.java:183) at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker= .java:79) at $Proxy6.getRowNum(Unknown Source) at org.apache.hadoop.hbase.client.coprocessor.AggregationClient$3.c= all(AggregationClient.java:220) at org.apache.hadoop.hbase.client.coprocessor.AggregationClient$3.c= all(AggregationClient.java:217) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImp= lementation$4.call(HConnectionManager.java:1463) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source= ) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Sourc= e) at java.lang.Thread.run(Unknown Source) Regards, Omkar Joshi -----Original Message----- From: Ted Yu [mailto:yuzhihong@gmail.com] Sent: Friday, April 19, 2013 3:00 PM To: user@hbase.apache.org Cc: user@hbase.apache.org Subject: Re: Speeding up the row count Since there is only one region in your table, using aggregation coprocessor= has no advantage. I think there may be some issue with your cluster - row count should finish= within 6 minutes. Have you checked server logs ? Thanks On Apr 19, 2013, at 12:33 AM, Omkar Joshi wro= te: > Hi, > > I'm having a 2-node(VMs) Hadoop cluster atop which HBase is running in th= e distributed mode. > > I'm having a table named ORDERS with >100000 rows. > > NOTE : Since my cluster is ultra-small, I didn't pre-split the table. > > ORDERS > rowkey : ORDER_ID > > column family : ORDER_DETAILS > columns : CUSTOMER_ID > PRODUCT_ID > REQUEST_DATE > PRODUCT_QUANTITY > PRICE > PAYMENT_MODE > > The java client code to simply check the count of the records is : > > public long getTableCount(String tableName, String columnFamilyName) { > > AggregationClient aggregationClient =3D new AggregationCli= ent(config); > Scan scan =3D new Scan(); > scan.addFamily(Bytes.toBytes(columnFamilyName)); > scan.setFilter(new FirstKeyOnlyFilter()); > > long rowCount =3D 0; > > try { > rowCount =3D aggregationClient.rowCount(Bytes.toBy= tes(tableName), > null, scan); > System.out.println("No. of rows in " + tableName += " is " > + rowCount); > } catch (Throwable e) { > // TODO Auto-generated catch block > e.printStackTrace(); > } > > return rowCount; > } > > It is running for more than 6 minutes now :( > > What shall I do to speed up the execution to milliseconds(at least a coup= le of seconds)? > > Regards, > Omkar Joshi > > > -----Original Message----- > From: Vedad Kirlic [mailto:kirlich@gmail.com] > Sent: Thursday, April 18, 2013 12:22 AM > To: user@hbase.apache.org > Subject: Re: Speeding up the row count > > Hi Omkar, > > If you are not interested in occurrences of specific column (e.g. name, > email ... ), and just want to get total number of rows (regardless of the= ir > content - i.e. columns), you should avoid adding any columns to the Scan,= in > which case coprocessor implementation for AggregateClient, will add > FirstKeyOnlyFilter to the Scan, so to avoid loading unnecessary columns, = so > this should result in some speed up. > > This is similar approach to what hbase shell 'count' implementation does, > although reduction in overhead in that case is bigger, since data transfe= r > from region server to client (shell) is minimized, whereas in case of > coprocessor, data does not leave region server, so most of the improvemen= t > in that case should come from avoiding loading of unnecessary files. Not > sure how this will apply to your particular case, given that data set per > row seems to be rather small. Also, in case of AggregateClient you will > benefit if/when your tables span multiple regions. Essentially, performan= ce > of this approach will 'degrade' as your table gets bigger, but only to th= e > point when it splits, from which point it should be pretty constant. Havi= ng > this in mind, and your type of data, you might consider pre-splitting you= r > tables. > > DISCLAIMER: this is mostly theoretical, since I'm not an expert in hbase > internals :), so your best bet is to try it - I'm too lazy to verify impa= ct > my self ;) > > Finally, if your case can tolerate eventual consistency of counters with > actual number of rows, you can, as already suggested, have RowCounter map > reduce run every once in a while, write the counter(s) back to hbase, and > read those when you need to obtain the number of rows. > > Regards, > Vedad > > > > -- > View this message in context: http://apache-hbase.679495.n3.nabble.com/Sp= eeding-up-the-row-count-tp4042378p4042415.html > Sent from the HBase User mailing list archive at Nabble.com. > > The contents of this e-mail and any attachment(s) may contain confidentia= l or privileged information for the intended recipient(s). Unintended recip= ients are prohibited from taking action on the basis of information in this= e-mail and using or disseminating the information, and must notify the s= ender and delete it from their system. L&T Infotech will not accept respons= ibility or liability for the accuracy or completeness of, or the presence o= f any virus or disabling code in this e-mail"