hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Colestock ...@ramblingredneck.com>
Subject Re: Hive 0.11 with Cloudera CHD4.3 MR v1
Date Fri, 23 Aug 2013 17:43:45 GMT
I miss the bit in the middle about adding the value to hive-site.xml (did the skip ahead bit)..
So after adding: 

<property>
<name>yarn.resourcemanager.address</name>
<value>foobar</value>
</property>

to hive-site.xml,  It's working!   Thanks so much for your help.. 
 
have a great weekend.. 

JC

 


On Aug 23, 2013, at 1:37 PM, Justin Workman <justinjworkman@gmail.com> wrote:

> I just went through this same process again and deployed to another 5 node cluster, and
it is also working as expected. I have ran anything very intensive yet, but did spawn several
mapreduce jobs.
> 
> 
> On Fri, Aug 23, 2013 at 11:11 AM, Jim Colestock <jc@ramblingredneck.com> wrote:
> Hello Justin, 
> 
> Thanks so much for the write up..  I'm compiling now.. I did find one error in your instructions:

> 
> patch -p0 > HIVE-4619.D10971.1.path
> 
> I believe it should be: 
> 
> patch -p0 < HIVE-4619.D10971.1.patch
> 
> 
> But alas, I still doesn't work for me.  I managed to get it to compile after using maven
to install a zookeeper dependency.. But when I try to do anything in hive that requires an
M/R job, I get the following: 
> 
> 2013-08-23 13:07:07,865 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing
logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-08-23 13:07:07,867 WARN org.apache.hadoop.mapred.Child: Error running child
> 
> java.lang.RuntimeException: java.lang.NullPointerException
>  at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
>  at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
>  at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
>  at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
>  at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
>  at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:396)
>  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>  at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.lang.NullPointerException
>  at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
>  ... 11 more
> 2013-08-23 13:07:07,871 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the
task
> 
> Any thoughts? 
> 
> Thanks again, 
> 
> JC
> 
> 
> 
> 
> 
> 
> 
> On Aug 22, 2013, at 11:15 PM, Justin Workman <justinjworkman@gmail.com> wrote:
> 
>> For anyone interested, I have finished my blog on getting Hive-0.11.0 working with
CDH4.3.0. It is a hack and can be made better but works like a charm.
>> 
>> http://www.justinjworkman.com/big-data/hive-0-11-0-on-cloudera/
>> 
>> 
>> 
>> On Thu, Aug 22, 2013 at 8:29 AM, Jim Colestock <jc@ramblingredneck.com> wrote:
>> Justin, 
>> 
>> That would be much appreciated! 
>> 
>> Thanks Again, 
>> 
>> JC
>> 
>> On Aug 22, 2013, at 9:18 AM, Justin Workman <justinjworkman@gmail.com> wrote:
>> 
>>> I do have this working on CHD4.3.0 with MR1 and Kerberos.  I am finishing up
a blog today explaining what was needed to get it working. I will reply with the link later
today. 
>>> 
>>> Justin 
>>> 
>>> Sent from my iPhone
>>> 
>>> On Aug 22, 2013, at 6:17 AM, Jim Colestock <jc@ramblingredneck.com> wrote:
>>> 
>>>> Hello All, 
>>>> 
>>>> Sorry for not replying to the thread, but I never received any of your responses..
 Hopefully this info will help: 
>>>> 
>>>> @kulkarni.swarnim
>>>> 
>>>> Here is the error I'm getting on task tracker for the mapper: 
>>>> 
>>>> 2013-08-21 15:34:32,491 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>>>> 2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running
child
>>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>> 	at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
>>>> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
>>>> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
>>>> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
>>>> 	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
>>>> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
>>>> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
>>>> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>>> 	at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>> 	at org.apache.hadoop.mapred.Child.main(Child.java:262)
>>>> Caused by: java.lang.NullPointerException
>>>> 	at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
>>>> 	... 11 more
>>>> 2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning cleanup
for the task
>>>> 
>>>> @Srinivas Surasani
>>>> 	Hopefully the above helps.. 
>>>> 
>>>> @Lars Francke
>>>> 	Thanks, I'll take a peek at those 2 patches.. But I've just recently pulled
down fresh code and recompiled everything and I'm still getting the above error. 
>>>> 
>>>> And for reference, below is my original email to the list..  Thanks all for
you help! 
>>>> 
>>>> 
>>>> JC
>>>> 
>>>> 
>>>> 
>>>> Hello All, 
>>>> 
>>>> Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3?  

>>>> 
>>>> I've been able to get hive to connect to my metadb (which is in Postgres).
 Verified by doing a show tables..  I can run explain and describes on tables, but when I
try to run anything that fires off an M/R job, I get the following error: 
>>>> 
>>>> hive>select count(*) from tableA;
>>>> Total MapReduce jobs = 1
>>>> Launching Job 1 out of 1
>>>> Number of reduce tasks determined at compile time: 1
>>>> In order to change the average load for a reducer (in bytes):
>>>>   set hive.exec.reducers.bytes.per.reducer=<number>
>>>> In order to limit the maximum number of reducers:
>>>>   set hive.exec.reducers.max=<number>
>>>> In order to set a constant number of reducers:
>>>>   set mapred.reduce.tasks=<number>
>>>> Starting Job = job_201307112247_13816, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>>>> Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_201307112247_13816
>>>> Hadoop job information for Stage-1: number of mappers: 1; number of reducers:
1
>>>> 2013-07-12 02:11:42,829 Stage-1 map = 0%,  reduce = 0%
>>>> 2013-07-12 02:12:08,173 Stage-1 map = 100%,  reduce = 100%
>>>> Ended Job = job_201307112247_13816 with errors
>>>> Error during job, obtaining debugging information...
>>>> Job Tracking URL: http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>>>> Examining task ID: task_201307112247_13816_m_000002 (and more) from job job_201307112247_13816
>>>> Exception in thread "Thread-19" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/util/HostUtil
>>>> at org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61)
>>>> at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
>>>> at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.util.HostUtil
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>> ... 4 more
>>>> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>>>> MapReduce Jobs Launched:
>>>> Job 0: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
>>>> Total MapReduce CPU Time Spent: 0 msec
>>>> 
>>>> I'm using my configs from hive 0.10, which works with no issues and this
was pretty much a drop in replacement on the machine that hadoop 0.10 was running on.. 
>>>> 
>>>> I've done a bunch of googling around and have found a bunch of other folks
that have have had the same issue, but no solid answers.. 
>>>> 
>>>> Thanks in advance for any help.. 
>>>> 
>>>> JC
>>>> 
>>>> 
>> 
>> 
> 
> 


Mime
View raw message