Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 664E693C6 for ; Thu, 8 Dec 2011 09:28:05 +0000 (UTC) Received: (qmail 12617 invoked by uid 500); 8 Dec 2011 09:28:03 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 12532 invoked by uid 500); 8 Dec 2011 09:27:56 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 12524 invoked by uid 99); 8 Dec 2011 09:27:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Dec 2011 09:27:54 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of stutiawasthi@hcl.com designates 203.105.185.23 as permitted sender) Received: from [203.105.185.23] (HELO gws05.hcl.com) (203.105.185.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Dec 2011 09:27:50 +0000 Received: from NDA-HCLIN-HT01.CORP.HCL.IN (10.248.64.35) by NDA-HCLIN-EDGE3.hcl.in (10.248.64.140) with Microsoft SMTP Server id 8.2.254.0; Thu, 8 Dec 2011 14:58:32 +0530 Received: from NDA-HCLC-HT01.HCLC.CORP.HCL.IN (10.33.64.155) by NDA-HCLIN-HT01.CORP.HCL.IN (10.248.64.35) with Microsoft SMTP Server (TLS) id 8.2.254.0; Thu, 8 Dec 2011 14:57:27 +0530 Received: from NDA-HCLC-EVS04.HCLC.CORP.HCL.IN ([10.33.64.194]) by NDA-HCLC-HT01.HCLC.CORP.HCL.IN ([::1]) with mapi; Thu, 8 Dec 2011 14:57:27 +0530 From: Stuti Awasthi To: "user@hbase.apache.org" Date: Thu, 8 Dec 2011 14:57:25 +0530 Subject: RE: CopyTable to remote cluster runs OK but doesn't copy anything Thread-Topic: CopyTable to remote cluster runs OK but doesn't copy anything Thread-Index: Acy1C7Oqe2VS8cX9R46Pe1Y92oCEygAduaYAAAIt/lA= Message-ID: <7D9AF4B98807C54EBADEA75DF6D5ACB7E32FF65F@NDA-HCLC-EVS04.HCLC.CORP.HCL.IN> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Hi, I have also tried copyTable with different clusters. It worked for me fine.= I set the hbase.zookeeper.quorum property in Hbase conf file. I used Hadoo= p-0.20.2. Thanks -----Original Message----- From: Jorn Argelo - Ephorus [mailto:Jorn.Argelo@ephorus.com] Sent: Thursday, December 08, 2011 2:29 PM To: user@hbase.apache.org Subject: RE: CopyTable to remote cluster runs OK but doesn't copy anything Hi Jon / J-D, Yeah, I had a bunch of additional stuff in my classpath which we needed for= other M/R jobs: /etc/zookeeper:/etc/hadoop-0.20/conf:/usr/lib/hadoop-0.20/*:/usr/lib/had oop-0.20/lib/*:/usr/lib/zookeeper/*:/usr/lib/zookeeper/lib/* I tried just removing /etc/zookeeper from the classpath but then I still ha= d the same result. After removing that whole line from the classpath I ende= d up with a working CopyTable. I could see that the MapReduce job was now c= aching jars in /tmp which it didn't do before. Maybe it's worthwhile to add this info the HBASE-4614? Let me know if there= 's any way I can assist with testing. Thanks a lot for your support. Jorn -----Oorspronkelijk bericht----- Van: Jonathan Hsieh [mailto:jon@cloudera.com] Verzonden: woensdag 7 december 2011 19:09 Aan: user@hbase.apache.org Onderwerp: Re: CopyTable to remote cluster runs OK but doesn't copy anythin= g Jorn, I recently ran into this problem. The CopyTable it actually is copying dat= a to the same instance of the table, and likely because an hbase client in = the MR job is picking up the settings from a zoo.cfg file. Have you added `hbase classpath` to your hadoop-env.sh file? Can you check= if zoo.cfg (possibly as /etc/zookeeper/* in CDH) is in the class path of = the task trackers.. If it is, you may want to remove it from there and then add the ZK settings= to your hbase-site.conf file. Jon. On Wed, Dec 7, 2011 at 9:31 AM, Jean-Daniel Cryans wrote: > It would most likely be this bug: > https://issues.apache.org/jira/browse/HBASE-4614 > > On Wed, Dec 7, 2011 at 12:27 AM, Jorn Argelo - Ephorus > wrote: > > Hi all, > > > > > > > > I'm trying to copy a table from one cluster to another cluster but this > > does not seem to do what I expect it to do. The Map/Reduce job runs > > successfully as you can see below, but it's not actually copying > > anything to the remote cluster. It almost looks as if it's not parsing > > the --peer.adr option and just copies the data inside the same cluster. > > At least, the "WARN mapred.JobClient: Use GenericOptionsParser for > > parsing the arguments. Applications should implement Tool for the same" > > warning would suggest that. > > > > > > > > Both clusters are running CHD3U1 and are both fully distributed, > > although hbase-test1 is a single physical server running all components > > for a fully distributed setup. The source cluster where I am running the > > job from is a small 10 node cluster. Note that on hbase-test1 the target > > table already exists with the same column families as in the source > > cluster. > > > > > > > > Does anybody have any idea what I'm doing wrong? Or maybe I found a bug? > > There's another guy at stackoverflow reporting the same issue > > (http://stackoverflow.com/questions/7952213/how-to-copy-a-table-from-one > > -hbase-cluster-to-another-cluster) but nobody responded on that. > > > > > > > > Thanks, > > > > Jorn > > > > > > > > > > > > $ hbase org.apache.hadoop.hbase.mapreduce.CopyTable > > --peer.adr=3Dhbase-test1:2181:/hbase chunk > > > > 11/12/07 08:52:24 WARN mapred.JobClient: Use GenericOptionsParser for > > parsing the arguments. Applications should implement Tool for the same. > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:zookeeper.version=3D3.3.3-cdh3u1--1, built on 07/18/2011 16:48 > > GMT > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:host.name=3Dnamenode1 > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:java.version=3D1.6.0_26 > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:java.vendor=3DSun Microsystems Inc. > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:java.home=3D/usr/lib/jvm/java-6-sun-1.6.0.26/jre > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:java.class.path=3D > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:java.library.path=3D/usr/lib/hbase/bin/../lib/native/Linux-amd > > 64-64:/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64 > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:java.io.tmpdir=3D/tmp > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:java.compiler=3D > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:os.name=3DLinux > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:os.arch=3Damd64 > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:os.version=3D2.6.32-33-server > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:user.name=3Dmapred > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:user.home=3D/usr/lib/hadoop-0.20 > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Client > > environment:user.dir=3D/usr/lib/hadoop-0.20 > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Initiating client > > connection, connectString=3Dhbase-test1:2181 sessionTimeout=3D10000 > > watcher=3Dhconnection > > > > 11/12/07 08:52:25 INFO zookeeper.ClientCnxn: Opening socket connection > > to server hbase-test1/10.30.10.10:2181 > > > > 11/12/07 08:52:25 INFO zookeeper.ClientCnxn: Socket connection > > established to hbase-test1/10.30.10.10:2181, initiating session > > > > 11/12/07 08:52:25 INFO zookeeper.ClientCnxn: Session establishment > > complete on server hbase-test1/10.30.10.10:2181, sessionid =3D > > 0x134126b2d250040, negotiated timeout =3D 10000 > > > > 11/12/07 08:52:25 DEBUG > > client.HConnectionManager$HConnectionImplementation: Lookedup root > > region location, > > connection=3Dorg.apache.hadoop.hbase.client.HConnectionManager$HConnection > > Implementation@105691e; hsa=3Dhbase-test1:60020 > > > > 11/12/07 08:52:25 DEBUG > > client.HConnectionManager$HConnectionImplementation: Cached location for > > .META.,,1.1028785192 is hbase-test1:60020 > > > > 11/12/07 08:52:25 DEBUG client.MetaScanner: Scanning .META. starting at > > row=3Dchunk,,00000000000000 for max=3D10 rows > > > > 11/12/07 08:52:25 DEBUG > > client.HConnectionManager$HConnectionImplementation: Cached location for > > chunk,,1323181597686.f527a21a31a39559a2f4cbd034d286a7. is > > hbase-test1:60020 > > > > 11/12/07 08:52:25 INFO mapreduce.TableOutputFormat: Created table > > instance for chunk > > > > 11/12/07 08:52:25 INFO zookeeper.ZooKeeper: Initiating client > > connection, connectString=3Dz01:2181,zk02:2181,zk03:2181 > > sessionTimeout=3D10000 watcher=3Dhconnection > > > > 11/12/07 08:52:25 INFO zookeeper.ClientCnxn: Opening socket connection > > to server zk02/10.30.4.93:2181 > > > > 11/12/07 08:52:25 INFO zookeeper.ClientCnxn: Socket connection > > established to zk02/10.30.4.93:2181, initiating session > > > > 11/12/07 08:52:25 INFO zookeeper.ClientCnxn: Session establishment > > complete on server zzk02/10.30.4.93:2181, sessionid =3D 0x233922a0b320c81, > > negotiated timeout =3D 10000 > > > > 11/12/07 08:52:25 DEBUG > > client.HConnectionManager$HConnectionImplementation: Lookedup root > > region location, > > connection=3Dorg.apache.hadoop.hbase.client.HConnectionManager$HConnection > > Implementation@3c3a1834; hsa=3Ddatanode1:60020 > > > > 11/12/07 08:52:25 DEBUG > > client.HConnectionManager$HConnectionImplementation: Cached location for > > .META.,,1.1028785192 is datanode1:60020 > > > > 11/12/07 08:52:25 DEBUG client.MetaScanner: Scanning .META. starting at > > row=3Dchunk,,00000000000000 for max=3D10 rows > > > > 11/12/07 08:52:25 DEBUG > > client.HConnectionManager$HConnectionImplementation: Cached location for > > chunk,,1323179990451.a243a485325744b9eedd8da2106712b6. is > > datanode3:60020 > > > > 11/12/07 08:52:25 DEBUG > > client.HConnectionManager$HConnectionImplementation: Cached location for > > chunk,array_for_lithiumion_battery_anode_material,1323179990451.6161771e > > cadc7a45acd28afbfca88a09. is datanode3:60020 > > > > 11/12/07 08:52:25 DEBUG > > client.HConnectionManager$HConnectionImplementation: Cached location for > > chunk,cytotoxic_elements_were_leached_from_the,1323179991855.5cf99f0425d > > 9e0e9fd41ac9645d65b93. is datanode3:60020 > > > > 11/12/07 08:52:25 DEBUG > > client.HConnectionManager$HConnectionImplementation: Cached location for > > chunk,generation_cephalosporin_just_before_biopsy,1323179991855.db0d0df2 > > c2e076bf01c75ae6ac200436. is datanode3:60020 > > > > 11/12/07 08:52:25 DEBUG > > client.HConnectionManager$HConnectionImplementation: Cached location for > > chunk,linked_with_the_2008_great_wenchuan,1323179964329.32ee9e359e50582b > > f1b419396c9aa8ad. is datanode2:60020 > > > > 11/12/07 08:52:25 DEBUG > > client.HConnectionManager$HConnectionImplementation: Cached location for > > chunk,pca_with_gabor_decomposition_offered_several,1323179964329.ed0b5be > > c77229df3b6a5a08c117db355. is datanode2:60020 > > > > 11/12/07 08:52:25 DEBUG > > client.HConnectionManager$HConnectionImplementation: Cached location for > > chunk,see_34\xE2\x80\x93_43however_when_many_partial,1323179993570.cf10c > > 9b3a05b1b26437c674af2b61cfc. is datanode3:60020 > > > > 11/12/07 08:52:25 DEBUG > > client.HConnectionManager$HConnectionImplementation: Cached location for > > chunk,the_wheelsground_contact_between_vehicle_tire,1323179993570.ee5baf > > fa6cdce4ced50c6a9c10beca75. is datanode3:60020 > > > > 11/12/07 08:52:25 DEBUG client.MetaScanner: Scanning .META. starting at > > row=3Dchunk,,00000000000000 for max=3D2147483647 rows > > > > 11/12/07 08:52:25 DEBUG mapreduce.TableInputFormatBase: getSplits: split > > -> 0 -> datanode3:,array_for_lithiumion_battery_anode_material > > > > 11/12/07 08:52:25 DEBUG mapreduce.TableInputFormatBase: getSplits: split > > -> 1 -> > > datanode3:array_for_lithiumion_battery_anode_material,cytotoxic_elements > > _were_leached_from_the > > > > 11/12/07 08:52:25 DEBUG mapreduce.TableInputFormatBase: getSplits: split > > -> 2 -> > > datanode3:cytotoxic_elements_were_leached_from_the,generation_cephalospo > > rin_just_before_biopsy > > > > 11/12/07 08:52:25 DEBUG mapreduce.TableInputFormatBase: getSplits: split > > -> 3 -> > > datanode3:generation_cephalosporin_just_before_biopsy,linked_with_the_20 > > 08_great_wenchuan > > > > 11/12/07 08:52:25 DEBUG mapreduce.TableInputFormatBase: getSplits: split > > -> 4 -> > > datanode2:linked_with_the_2008_great_wenchuan,pca_with_gabor_decompositi > > on_offered_several > > > > 11/12/07 08:52:25 DEBUG mapreduce.TableInputFormatBase: getSplits: split > > -> 5 -> > > datanode2:pca_with_gabor_decomposition_offered_several,see_34\xE2\x80\x9 > > 3_43however_when_many_partial > > > > 11/12/07 08:52:25 DEBUG mapreduce.TableInputFormatBase: getSplits: split > > -> 6 -> > > datanode3:see_34\xE2\x80\x93_43however_when_many_partial,the_wheelsgroun > > d_contact_between_vehicle_tire > > > > 11/12/07 08:52:25 DEBUG mapreduce.TableInputFormatBase: getSplits: split > > -> 7 -> datanode3:the_wheelsground_contact_between_vehicle_tire, > > > > 11/12/07 08:52:25 INFO mapred.JobClient: Running job: > > job_201111021158_0026 > > > > 11/12/07 08:52:26 INFO mapred.JobClient: map 0% reduce 0% > > > > 11/12/07 08:56:22 INFO mapred.JobClient: map 12% reduce 0% > > > > 11/12/07 08:56:53 INFO mapred.JobClient: map 25% reduce 0% > > > > 11/12/07 08:59:05 INFO mapred.JobClient: map 37% reduce 0% > > > > 11/12/07 08:59:51 INFO mapred.JobClient: map 50% reduce 0% > > > > 11/12/07 09:00:31 INFO mapred.JobClient: map 62% reduce 0% > > > > 11/12/07 09:00:35 INFO mapred.JobClient: map 75% reduce 0% > > > > 11/12/07 09:00:43 INFO mapred.JobClient: map 87% reduce 0% > > > > 11/12/07 09:01:02 INFO mapred.JobClient: map 100% reduce 0% > > > > 11/12/07 09:01:03 INFO mapred.JobClient: Job complete: > > job_201111021158_0026 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: Counters: 13 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: Job Counters > > > > 11/12/07 09:01:03 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=3D3306288 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: Total time spent by all > > reduces waiting after reserving slots (ms)=3D0 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: Total time spent by all > > maps waiting after reserving slots (ms)=3D0 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: Rack-local map tasks=3D8 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: Launched map tasks=3D13 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: Data-local map tasks=3D5 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=3D0 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: FileSystemCounters > > > > 11/12/07 09:01:03 INFO mapred.JobClient: HDFS_BYTES_READ=3D1254 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: FILE_BYTES_WRITTEN=3D523502 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: Map-Reduce Framework > > > > 11/12/07 09:01:03 INFO mapred.JobClient: Map input records=3D26892941 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: Spilled Records=3D0 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: Map output records=3D26892941 > > > > 11/12/07 09:01:03 INFO mapred.JobClient: SPLIT_RAW_BYTES=3D1254 > > > > > > > > > > > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // jon@cloudera.com ::DISCLAIMER:: ---------------------------------------------------------------------------= -------------------------------------------- The contents of this e-mail and any attachment(s) are confidential and inte= nded for the named recipient(s) only. It shall not attach any liability on the originator or HCL or its affiliate= s. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect t= he opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,= distribution and / or publication of this message without the prior written consent of the author of this e-mail= is strictly prohibited. If you have received this email in error please delete it and notify the sender immedia= tely. Before opening any mail and attachments please check them for viruses and defect. ---------------------------------------------------------------------------= --------------------------------------------