hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vijay shinde <vijaysanj...@gmail.com>
Subject Re: HBASE and HIVE Integration
Date Thu, 26 Jul 2012 18:54:36 GMT
Hi Bejoy,

Thanks for your reply. My Hbase is working fine independently.
I was able reolve the zookeeper error. I had issues with hive-site.xml
configuration..

Now my job run little bit longer and this is the lastest error

I am on Cloudera CDH3U4 demo VM

Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207260911_0003, Tracking URL =
http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207260911_0003
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job  -Dmapred.job.tracker=
0.0.0.0:8021 -kill job_201207260911_0003
2012-07-26 13:34:55,900 Stage-0 map = 0%,  reduce = 0%
2012-07-26 13:35:56,207 Stage-0 map = 0%,  reduce = 0%
2012-07-26 13:36:56,364 Stage-0 map = 0%,  reduce = 0%
2012-07-26 13:37:57,342 Stage-0 map = 0%,  reduce = 0%
2012-07-26 13:38:57,398 Stage-0 map = 0%,  reduce = 0%
2012-07-26 13:39:57,467 Stage-0 map = 0%,  reduce = 0%
2012-07-26 13:40:57,852 Stage-0 map = 0%,  reduce = 0%
2012-07-26 13:41:58,724 Stage-0 map = 0%,  reduce = 0%
2012-07-26 13:42:59,700 Stage-0 map = 0%,  reduce = 0%
2012-07-26 13:44:00,582 Stage-0 map = 0%,  reduce = 0%
2012-07-26 13:45:00,977 Stage-0 map = 0%,  reduce = 0%
2012-07-26 13:46:01,110 Stage-0 map = 0%,  reduce = 0%
2012-07-26 13:46:05,443 Stage-0 map = 100%,  reduce = 100%
Ended Job = job_201207260911_0003 with errors

hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where
foo=98;

java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {"foo":98,"bar":"val_98"}
 at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
 at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row {"foo":98,"bar":"val_98"}
 at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
 at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
 ... 8 more
Caused by: or
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
hive>



On Thu, Jul 26, 2012 at 9:55 AM, Bejoy Ks <bejoy_ks@yahoo.com> wrote:

> Hi Vijay
>
> Is your hbase working independently without any issues. I mean, are you
> able to insert data into hbase tables without using hive integration?
>
> Was the same error message thrown when you directly provided hbase.master
> instead of zookeeper quorum?
>
> Regards
> Bejoy KS
>
>
> ________________________________
> From: Hezhiqiang (Ransom) <ransom.hezhiqiang@huawei.com>
> To: "user@hive.apache.org" <user@hive.apache.org>
> Cc: Zhaojun (Terry) <terry.zhaojun@huawei.com>
> Sent: Thursday, July 26, 2012 9:06 AM
> Subject: RE: HBASE and HIVE Integration
>
>
>
> Hi´╝îVijay
>          You need to add zookeeper.jar to the hive-site.xml
> <name>hive.aux.jars.path</name>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar,
> file:///usr/lib/hive/lib/ zookeeper-3.4.3.jar</value>
> </property>
> <name>hbase.zookeeper.quorum</name>
> <value>localhost</value>
> </property>
>
>
> Regards
> Ransom
>
> From:vijay shinde [mailto:vijaysanjyot@gmail.com]
> Sent: Thursday, July 26, 2012 8:59 AM
> To: user@hive.apache.org; Bejoy Ks
> Subject: Re: HBASE and HIVE Integration
>
> Hi Bejoy,
>
> I made some changes as per your suggetion.
>
> Here is the error from the
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004 Job:
>
> Error: java.lang.ClassNotFoundException:
> org.apache.zookeeper.KeeperException
>  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>  at sun.misc.Launcher$
>
> I went ahead and updated the hadoop-env.sh file and set the class path for
> hbase and zookeeper as follows:
> # Extra Java CLASSPATH elements.  Optional.
> export
> HADOOP_CLASSPATH="/usr/lib/hive/lib/hbase-0.92.jar:/usr/lib/hive/lib/zookeeper-3.4.3.jar:$HADOOP_CLASSPATH"
>
> Here is snippet of hive-site.xml file
>
> <name>hive.aux.jars.path</name>
>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar</value>
> </property>
> <name>hbase.zookeeper.quorum</name>
> <value>localhost</value>
> </property>
>
> Error message while executing hive query
>
> [root@localhost hive]# ./bin/hive
> Hive history file=/tmp/root/hive_job_log_root_201207252044_1993919630.txt
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where
> foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207251858_0004, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207251858_0004
> 2012-07-25 20:46:38,207 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 20:47:35,920 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207251858_0004 with errors
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> hive>
>
> I am lost need help badly!!!!!!!!!!!!!
>
> Vijay
>
>
>
> On Wed, Jul 25, 2012 at 9:47 AM, Bejoy Ks <bejoy_ks@yahoo.com> wrote:
> Hi Vijay
>
> You have provided the hbase master directly. (It is fine for single node
> hbase installation). But still can you try providing the zookeeper quorum
> instead.
>
> If that doesn't work as well , please post the error log from the
> mapreduce tasks?
>
> Just go the jobtracker page and drill down on the corresponding job to get
> the failed tasks. From each failed tasks you can get the error logs.
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
>
>
> Regards
> Bejoy KS
>
>
> ________________________________
>
> From:vijay shinde <vijaysanjyot@gmail.com>
> To:user@hive.apache.org; bejoy_ks@yahoo.com
> Sent: Wednesday, July 25, 2012 6:58 PM
> Subject: Re: HBASE and HIVE Integration
>
> Hi Bejoy,
>
> Thanks for quick reply. Here are some additional details
>
> Cloudera Version - CDH3U4
>
> hive-site.xml
>
>
> <property>
> <name>hive.aux.jars.path</name>
>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar,file:///usr/lib/hive/lib/hbase-0.90.4-cdh3u2.jar,file:///usr/lib/hive/lib/zookeeper-3.3.1.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar</value>
> </property>
> Execution Log
>
> 1. start zookeeper
> [root@localhost zookeeper]# ./bin/zkServer.sh start
>
> 2. start hbase
>
> 3. start hive. I am setting hive jars in hive-site.xml
>
> ./bin/hive -hiveconf hbase.master=127.0.1.1:60010
>
> 4. Create new HBase table which is to be managed by Hive
>
> CREATE TABLE hive_hbasetable_k(key int, value string)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
> TBLPROPERTIES ("hbase.table.name" = "hivehbasek");
>
> 5. Create a logical table pokes in Hive
> CREATE TABLE pokes (foo INT, bar STRING);
> 6. HIve error while inserting the data from Hive Poke table to HBASE table
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
> foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207250246_0005, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job  -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207250246_0005
> 2012-07-25 04:26:00,198 Stage-0 map = 0%,  reduce = 0%
> 2012-07-25 04:27:00,767 Stage-0 map = 0%,  reduce = 0%
> 2012-07-25 04:27:08,844 Stage-0 map = 100%,  reduce = 100%
> Ended Job = job_201207250246_0005 with errors
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> Let me know if you need any additonal information.
>
> Thanks,
> Vijay
> On Wed, Jul 25, 2012 at 5:30 AM, Bejoy KS <bejoy_ks@yahoo.com> wrote:
> Hi Vijay
>
> Can you share more details like
>
> The CDH Version/Hive version you are using
>
> Steps you followed for hive hbase integration with the values you set
>
> The DDL used for hive hbase integration
>
> The actual error from failed map reduce task
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
>
> ________________________________
>
> From: vijay shinde <vijaysanjyot@gmail.com>
> Date: Wed, 25 Jul 2012 04:45:41 -0400
> To: <user@hive.apache.org>
> ReplyTo: user@hive.apache.org
> Subject: HBASE and HIVE Integration
>
> I am facing issue while executing HIVE queries on HBASE-HIVE integration.
> I followed the wiki hbase-hive integration
> https://cwiki.apache.org/Hive/hbaseintegration.html
>
> I have already passed all the required jars for auxpath in hive-site.xml
> file.
> I am using Cloudera CDH demo VM.. Any help would be highly appreciated
>
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
> foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207250246_0005, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job  -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207250246_0005
> 2012-07-25 04:26:00,198 Stage-0 map = 0%,  reduce = 0%
> 2012-07-25 04:27:00,767 Stage-0 map = 0%,  reduce = 0%
> 2012-07-25 04:27:08,844 Stage-0 map = 100%,  reduce = 100%
> Ended Job = job_201207250246_0005 with errors
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> hive>
>

Mime
View raw message