Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EE31173A2 for ; Sat, 3 Dec 2011 07:32:40 +0000 (UTC) Received: (qmail 50428 invoked by uid 500); 3 Dec 2011 07:32:39 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 49583 invoked by uid 500); 3 Dec 2011 07:32:38 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 49561 invoked by uid 99); 3 Dec 2011 07:32:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Dec 2011 07:32:31 +0000 X-ASF-Spam-Status: No, hits=0.7 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [208.109.78.208] (HELO smtpoutwbe06.prod.mesa1.secureserver.net) (208.109.78.208) by apache.org (qpsmtpd/0.29) with SMTP; Sat, 03 Dec 2011 07:32:24 +0000 Received: (qmail 15156 invoked from network); 3 Dec 2011 07:32:01 -0000 Received: from unknown (HELO localhost) (72.167.218.143) by smtpoutwbe06.prod.mesa1.secureserver.net with SMTP; 3 Dec 2011 07:32:01 -0000 Received: (qmail 24138 invoked by uid 99); 3 Dec 2011 07:32:01 -0000 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" X-Originating-IP: 24.160.172.101 User-Agent: Web-Based Email 5.6.06 Message-Id: <20111203003200.7098ff60d8000ac98b0356cd27c73871.80c9def504.wbe@email04.secureserver.net> From: To: user@hive.apache.org Subject: RE: Hive-Hbase integration require Hbase in Pseudo distributed =?UTF-8?Q?=3F=3F?= Date: Sat, 03 Dec 2011 00:32:00 -0700 Mime-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org =0AGotcha. Thanks John. If I getting running I will let you know.=0A=0A=0A = -------- Original Message --------=0A Subject: Re: Hive-Hbase integration = require Hbase in Pseudo=0A distributed??=0A From: John Sichi =0A Date: Fri, December 02, 2011 4:51 pm=0A To: "" <= user@hive.apache.org>=0A =0A As you can guess from the 0.89 dependency, the= re has been a lot of=0Awater under the bridge since this integration was de= veloped. If someone=0Awould like to take on bringing it up to date, that wo= uld be great.=0A =0A Note that auxpath is to make the jars available in map= /reduce task VM's=0A(we don't put everything from lib there automatically).= =0A =0A JVS=0A =0A On Dec 2, 2011, at 10:39 AM, = =0A wrote:=0A =0A > =0A > I am having the same issue. Hive won't connect to= HBase and throws=0A > org.apache.hadoop.hbase.MasterNotRunningException de= spite the fact=0Athat=0A > the master is up and running. It may only work i= f HBase is in=0A > distributed mode or psuedo-distributed mode. I know HBas= e doesn't put=0A > files into HDFS otherwise.=0A > =0A > =0A > It certainly= doesn't work for me running in standalone mode. I've=0Atried=0A > about th= irty different combinations of hive/hbase and can't get it=0Agoing=0A > on = any of them, so I switched to trying to get pseudo-distributed=0Amode=0A > = working in HBase, but haven't been able to find the magic combination=0Aof= =0A > versions that will allow HBase to do anything in HDFS other than=0Ath= row=0A > EOFExceptions.=0A > =0A > In any case, according to the Hive docum= entation (see below) it=0Adoesn't=0A > work with any version of HBase other= than 0.89, but there are three=0A0.89=0A > versions of HBase at archive.ap= ache.org and the lib directories for=0AHive=0A > contain 0.89-SNAPSHOT.=0A = > =0A > =0A > FYI: There's an official Hive/HBase integration page at the= =0AConfluence=0A > wiki, but that doesn't work either:=0A > =0A > https://c= wiki.apache.org/confluence/display/Hive/HBaseIntegration=0A > =0A > =0A > I= t contains the instruction:=0A > =0A > =0A > "The handler requires Hadoop 0= .20 or higher, and has only been tested=0A > with dependency versions hadoo= p-0.20.x, hbase-0.89.0 and=0A > zookeeper-3.3.1. If you are not using hbase= -0.89.0, you will need to=0A > rebuild the handler with the HBase jar match= ing your version, and=0Achange=0A > the --auxpath above accordingly. Failur= e to use matching versions=0Awill=0A > lead to misleading connection failur= es such as =0A > =0A > MasterNotRunningException =0A > =0A > since the HBas= e RPC protocol changes often."=0A > =0A > =0A > But that really doesn't mak= e sense. Hive uses Ivy and you can't just=0A > simply replace the jar files= . Updating the version Ivy fetches from=0Athe=0A > Apache repository in the= second sentence contradicts the version=0A > exception in the previous sen= tence since the only official releases=0Aof=0A > HBase are 0.90 and forward= :=0A > =0A >=0Ahttps://repository.apache.org/content/repositories/releases/= org/apache/hbase/hbase/=0A > =0A > I'm trying to get Hive to build against = HBase 0.90 but Ivy wants to=0Apull=0A > the 0.90 out of snapshots so trying= to grab the jar file throws 404s.=0A > =0A > =0A > As a side note: the --a= uxpath seems unnecessary. The jars are already=0Ain=0A > the lib directory = so it seems like they ought to be on the classpath=0A > already. =0A > =0A = > =0A > =0A > -------- Original Message --------=0A > Subject: Re: Hive-Hba= se integration require Hbase in Pseudo=0A > distributed??=0A > From: Mohamm= ad Tariq =0A > Date: Fri, December 02, 2011 7:28 am=0A = > To: user@hive.apache.org=0A > =0A > Anyone there, cud you please confirm = if I can use hive-hbase in=0A > standalone mode???=0A > will it work? or sh= ould i go for Pseudo distributed mode ?=0A > =0A > Regards,=0A > Mohammad T= ariq=0A > =0A > =0A > =0A > On Fri, Dec 2, 2011 at 5:54 PM, Alok Kumar wrote:=0A >> hi,=0A >> =0A >> yeah i've used=0A >> =0A >> $= HIVE_HOME/bin/hive --auxpath=0A >>=0A$HIVE_HOME/lib/hive-hbase-handler-*.ja= r,$HIVE_HOME/lib/hbase-*.jar,$HIVE_HOME/lib/zookeeper-*.jar=0A >> -hiveconf= hbase.master=3Dlocalhost:60000=0A >> -------------------------------------= -----------=0A >> =0A >> Hadoop version : hadoop-0.20.203.0=0A >> Hbase ver= sion : hbase-0.90.4=0A >> Hive version : hive-0.9.0 (built from trunk)=0A >= > on=0A >> Ubuntu 11.10=0A >> ---------------------------------------------= --=0A >> =0A >> Regards,=0A >> =0A >> Alok=0A >> =0A >> =0A >> On Fri, Dec = 2, 2011 at 5:49 PM, Ankit Jain =0Awrote:=0A >>> = =0A >>> Hi,=0A >>> =0A >>> have you used following command to start the hiv= e shell.=0A >>> =0A >>> $HIVE_HOME/bin/hive --auxpath=0A >>>=0A$HIVE_HOME/l= ib/hive-hbase-handler-*.jar,$HIVE_HOME/lib/hbase-*.jar,$HIVE_HOME/lib/zooke= eper-*.jar=0A >>> -hiveconf hbase.master=3D127.0.0.1:60000=0A >>> =0A >>> = =0A >>> If no then used above command.=0A >>> Regards,=0A >>> Ankit=0A >>> = =0A >>> =0A >>> On Fri, Dec 2, 2011 at 5:34 PM, Alok Kumar =0Awrote:=0A >>>> =0A >>>> Hi,=0A >>>> =0A >>>> // Hadoop core-site.xml= =0A >>>> =0A >>>> =0A >>>> fs.default.name=0A >>>> hdfs://localhost:9000=0A >>>> =0A >= >>> =0A >>>> hadoop.tmp.dir=0A >>>> /home/alo= kkumar/hadoop/tmp=0A >>>> =0A >>>> =0A >= >>> =0A >>>> // hbase-site.xml=0A >>>> =0A >>>> = =0A >>>> hbase.rootdir=0A >>>> =0A >>>> file:///home/alokkumar/hbase/= =0A >>>> =0A >>>> =0A >>>> =0A >>>> with these c= onf Hbase/Hive are runnning independently file..=0A >>>> hbase(main):003:0>= status=0A >>>> 1 servers, 0 dead, 4.0000 average load=0A >>>> =0A >>>> but= i'm stil getting $hive> CREATE TABLE hbase_table_1(key int,=0Avalue=0A >>>= > string)=0A >>>>> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHand= ler'=0A >>>>> WITH SERDEPROPERTIES ("hbase.columns.mapping" =3D ":key,cf1:v= al");=0A >>>> FAILED: Error in metadata:=0A >>>>=0AMetaException(message:or= g.apache.hadoop.hbase.MasterNotRunningException:=0A >>>> localhost:45966=0A= >>>> =0A >>>> at=0A >>>>=0Aorg.apache.hadoop.hbase.client.HConnectionManag= er$HConnectionImplementation.getMaster(HConnectionManager.java:394)=0A >>>>= at=0A >>>>=0Aorg.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.j= ava:83)=0A >>>> =0A >>>> Regards,=0A >>>> Alok=0A >>>> =0A >>>> =0A >>>> On= Fri, Dec 2, 2011 at 5:14 PM, Ankit Jain=0A=0A >>>= > wrote:=0A >>>>> =0A >>>>> HI,=0A >>>>> =0A >>>>> Can you post the Hbase-s= ite.xml and hadoop core-site.xml=0Aproperties=0A >>>>> here.=0A >>>>> =0A >= >>>> Regards,=0A >>>>> Ankit=0A >>>>> =0A >>>>> =0A >>>>> On Fri, Dec 2, 20= 11 at 3:30 PM, Alok Kumar =0Awrote:=0A >>>>>> =0A >>>>>>= Hi Ankit,=0A >>>>>> =0A >>>>>> you were right, my Hbase shell/HMaster was = not running though it=0Awas=0A >>>>>> coming in jps :)=0A >>>>>> =0A >>>>>>= nw i've run my HMaster n Hbase shell is up.. n getting this=0Aerror--=0A >= >>>>> Do I need zookeeper configured in standalone mode?=0A >>>>>> =0A >>>>= >> hive> CREATE TABLE hbase_table_1(key int, value string)=0A >>>>>>> STORE= D BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'=0A >>>>>>> WITH SER= DEPROPERTIES ("hbase.columns.mapping" =3D ":key,cf1:val")=0A >>>>>>> TBLPRO= PERTIES ("hbase.table.name" =3D "xyz");=0A >>>>>> =0A >>>>>> FAILED: Error = in metadata:=0A >>>>>>=0AMetaException(message:org.apache.hadoop.hbase.ZooK= eeperConnectionException:=0A >>>>>> org.apache.hadoop.hbase.ZooKeeperConnec= tionException:=0A >>>>>> org.apache.zookeeper.KeeperException$ConnectionLos= sException:=0A >>>>>> KeeperErrorCode =3D ConnectionLoss for /hbase=0A >>>>= >> at=0A >>>>>>=0Aorg.apache.hadoop.hbase.client.HConnectionManager$HConnec= tionImplementation.getZooKeeperWatcher(HConnectionManager.java:985)=0A >>>>= >> at=0A >>>>>>=0Aorg.apache.hadoop.hbase.client.HConnectionManager$HConnec= tionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)=0A >= >>>>> at=0A >>>>>>=0Aorg.apache.hadoop.hbase.client.HConnectionManager$HCon= nectionImplementation.(HConnectionManager.java:292)=0A >>>>>> at=0A >= >>>>>=0Aorg.apache.hadoop.hbase.client.HConnectionManager.getConnection(HCo= nnectionManager.java:155)=0A >>>>>> at=0A >>>>>>=0Aorg.apache.hadoop.hbase.= client.HBaseAdmin.(HBaseAdmin.java:79)=0A >>>>>> =0A >>>>>> at=0A >>>= >>>=0Aorg.apache.hadoop.hive.hbase.HBaseStorageHandler.getHBaseAdmin(HBaseS= torageHandler.java:74)=0A >>>>>> at=0A >>>>>>=0Aorg.apache.hadoop.hive.hbas= e.HBaseStorageHandler.preCreateTable(HBaseStorageHandler.java:158)=0A >>>>>= > at=0A >>>>>>=0Aorg.apache.hadoop.hive.metastore.HiveMetaStoreClient.creat= eTable(HiveMetaStoreClient.java:375)=0A >>>>>> at=0A >>>>>>=0Aorg.apache.ha= doop.hive.ql.metadata.Hive.createTable(Hive.java:540)=0A >>>>>> at=0A >>>>>= >=0Aorg.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3473)= =0A >>>>>> at=0Aorg.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java= :225)=0A >>>>>> at=0Aorg.apache.hadoop.hive.ql.exec.Task.executeTask(Task.j= ava:133)=0A >>>>>> at=0A >>>>>>=0Aorg.apache.hadoop.hive.ql.exec.TaskRunner= .runSequential(TaskRunner.java:57)=0A >>>>>> at org.apache.hadoop.hive.ql.D= river.launchTask(Driver.java:1332)=0A >>>>>> at org.apache.hadoop.hive.ql.D= river.execute(Driver.java:1123)=0A >>>>>> at org.apache.hadoop.hive.ql.Driv= er.run(Driver.java:931)=0A >>>>>> at=0A >>>>>>=0Aorg.apache.hadoop.hive.cli= .CliDriver.processLocalCmd(CliDriver.java:255)=0A >>>>>> at=0A >>>>>>=0Aorg= .apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)=0A >>>>>> = at=0A >>>>>>=0Aorg.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.j= ava:403)=0A >>>>>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.ja= va:671)=0A >>>>>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.ja= va:554)=0A >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Me= thod)=0A >>>>>> at=0A >>>>>>=0Asun.reflect.NativeMethodAccessorImpl.invoke(= NativeMethodAccessorImpl.java:39)=0A >>>>>> at=0A >>>>>>=0Asun.reflect.Dele= gatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)=0A >>= >>>> at java.lang.reflect.Method.invoke(Method.java:597)=0A >>>>>> at org.a= pache.hadoop.util.RunJar.main(RunJar.java:156)=0A >>>>>> Caused by: org.apa= che.hadoop.hbase.ZooKeeperConnectionException:=0A >>>>>> org.apache.zookeep= er.KeeperException$ConnectionLossException:=0A >>>>>> KeeperErrorCode =3D C= onnectionLoss for /hbase=0A >>>>>> at=0A >>>>>>=0Aorg.apache.hadoop.hbase.z= ookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:147)=0A >>>>>> at=0A= >>>>>>=0Aorg.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl= ementation.getZooKeeperWatcher(HConnectionManager.java:983)=0A >>>>>> ... 2= 5 more=0A >>>>>> Caused by:=0A >>>>>> org.apache.zookeeper.KeeperException$= ConnectionLossException:=0A >>>>>> KeeperErrorCode =3D ConnectionLoss for /= hbase=0A >>>>>> at=0A >>>>>>=0Aorg.apache.zookeeper.KeeperException.create(= KeeperException.java:90)=0A >>>>>> at=0A >>>>>>=0Aorg.apache.zookeeper.Keep= erException.create(KeeperException.java:42)=0A >>>>>> at org.apache.zookeep= er.ZooKeeper.create(ZooKeeper.java:637)=0A >>>>>> at=0A >>>>>>=0Aorg.apache= .hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:886)=0A >>>>= >> at=0A >>>>>>=0Aorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.= (ZooKeeperWatcher.java:133)=0A >>>>>> ... 26 more=0A >>>>>> =0A >>>>>> )=0A= >>>>>> FAILED: Execution Error, return code 1 from=0A >>>>>> org.apache.ha= doop.hive.ql.exec.DDLTask=0A >>>>>> =0A >>>>>> =0A >>>>>> On Fri, Dec 2, 20= 11 at 2:03 PM, Ankit Jain=0A=0A >>>>>> wrote:=0A >= >>>>>> =0A >>>>>>> I think your hbase master is not running.=0A >>>>>>> =0A= >>>>>>> Open the hive shell and run command :=0A >>>>>>> hbase> status=0A = >>>>>>> =0A >>>>>>> =0A >>>>>>> =0A >>>>>>> On Fri, Dec 2, 2011 at 2:00 PM,= Alok Kumar =0Awrote:=0A >>>>>>>> =0A >>>>>>>> Hi,=0A >>= >>>>>> =0A >>>>>>>> Does Hive-Hbase integration require Hbase running in=0A= >>>>>>>> pseudo-distributed mode?=0A >>>>>>>> =0A >>>>>>>> I've build my H= adoop following this article=0A >>>>>>>>=0Ahttp://www.michael-noll.com/blog= /2011/04/14/building-an-hadoop-0-20-x-version-for-hbase-0-90-2/=0A >>>>>>>>= and have already replaced Hbase jar files accordingly..=0A >>>>>>>> =0A >>= >>>>>> I'm getting this error..=0A >>>>>>>> =0A >>>>>>>> hive> !jps;=0A >>>= >>>>> 5469 Jps=0A >>>>>>>> 4128 JobTracker=0A >>>>>>>> 3371 Main=0A >>>>>>>= > 4346 TaskTracker=0A >>>>>>>> 5330 RunJar=0A >>>>>>>> 4059 SecondaryNameNo= de=0A >>>>>>>> 8350 NameNode=0A >>>>>>>> 3841 DataNode=0A >>>>>>>> 3244 HMa= ster=0A >>>>>>>> hive> create table hbase_table_1(key int, value string) st= ored=0Aby=0A >>>>>>>> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' wi= th=0A >>>>>>>> serdeproperties("hbase.columns.mapping" =3D ":key,cf1:val")= =0Atblproperties=0A >>>>>>>> ("hbase.table.name" =3D "xyz");=0A >>>>>>>> = =0A >>>>>>>> FAILED: Error in metadata:=0A >>>>>>>>=0AMetaException(message= :org.apache.hadoop.hbase.MasterNotRunningException:=0A >>>>>>>> localhost:5= 6848=0A >>>>>>>> at=0A >>>>>>>>=0Aorg.apache.hadoop.hbase.client.HConnectio= nManager$HConnectionImplementation.getMaster(HConnectionManager.java:394)= =0A >>>>>>>> at=0A >>>>>>>>=0Aorg.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:83)=0A >>>>>>>> at=0A >>>>>>>>=0Aorg.apache.hadoop.hive= .hbase.HBaseStorageHandler.getHBaseAdmin(HBaseStorageHandler.java:74)=0A >>= >>>>>> at=0A >>>>>>>>=0Aorg.apache.hadoop.hive.hbase.HBaseStorageHandler.pr= eCreateTable(HBaseStorageHandler.java:158)=0A >>>>>>>> at=0A >>>>>>>>=0Aorg= .apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStore= Client.java:375)=0A >>>>>>>> at=0A >>>>>>>>=0Aorg.apache.hadoop.hive.ql.met= adata.Hive.createTable(Hive.java:540)=0A >>>>>>>> at=0A >>>>>>>>=0Aorg.apac= he.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3473)=0A >>>>>>>> a= t=0A >>>>>>>>=0Aorg.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java= :225)=0A >>>>>>>> at=0Aorg.apache.hadoop.hive.ql.exec.Task.executeTask(Task= .java:133)=0A >>>>>>>> at=0A >>>>>>>>=0Aorg.apache.hadoop.hive.ql.exec.Task= Runner.runSequential(TaskRunner.java:57)=0A >>>>>>>> at=0Aorg.apache.hadoop= .hive.ql.Driver.launchTask(Driver.java:1332)=0A >>>>>>>> at org.apache.hado= op.hive.ql.Driver.execute(Driver.java:1123)=0A >>>>>>>> at org.apache.hadoo= p.hive.ql.Driver.run(Driver.java:931)=0A >>>>>>>> at=0A >>>>>>>>=0Aorg.apac= he.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)=0A >>>>>>>= > at=0A >>>>>>>>=0Aorg.apache.hadoop.hive.cli.CliDriver.processCmd(CliDrive= r.java:212)=0A >>>>>>>> at=0A >>>>>>>>=0Aorg.apache.hadoop.hive.cli.CliDriv= er.processLine(CliDriver.java:403)=0A >>>>>>>> at=0Aorg.apache.hadoop.hive.= cli.CliDriver.run(CliDriver.java:671)=0A >>>>>>>> at=0Aorg.apache.hadoop.hi= ve.cli.CliDriver.main(CliDriver.java:554)=0A >>>>>>>> at sun.reflect.Native= MethodAccessorImpl.invoke0(Native Method)=0A >>>>>>>> at=0A >>>>>>>>=0Asun.= reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)= =0A >>>>>>>> at=0A >>>>>>>>=0Asun.reflect.DelegatingMethodAccessorImpl.invo= ke(DelegatingMethodAccessorImpl.java:25)=0A >>>>>>>> at java.lang.reflect.M= ethod.invoke(Method.java:597)=0A >>>>>>>> at org.apache.hadoop.util.RunJar.= main(RunJar.java:156)=0A >>>>>>>> )=0A >>>>>>>> FAILED: Execution Error, re= turn code 1 from=0A >>>>>>>> org.apache.hadoop.hive.ql.exec.DDLTask=0A >>>>= >>>> =0A >>>>>>>> where i shuld look for fixing=0A >>>>>>>> "message:org.ap= ache.hadoop.hbase.MasterNotRunningException:=0Alocalhost:56848"=0A >>>>>>>>= ..?=0A >>>>>>>> =0A >>>>>>>> --=0A >>>>>>>> regards=0A >>>>>>>> Alok Kumar= =0A >>>>>>>> =0A >>>>>>>> =0A >>>>>>> =0A >>>>>> --=0A >>>>>> Alok=0A >>>>>= > =0A >>>>>> =0A >>>>> =0A >>>> =0A >>>> =0A >>> =0A >> =0A >> =0A >> =0A >= > --=0A >> Alok Kumar=0A >> =0A >> =0A >=0A