hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Justin Workman <justinjwork...@gmail.com>
Subject Re: Hive 0.11 with Cloudera CHD4.3 MR v1
Date Fri, 23 Aug 2013 03:15:00 GMT
For anyone interested, I have finished my blog on getting Hive-0.11.0
working with CDH4.3.0. It is a hack and can be made better but works like a
charm.

http://www.justinjworkman.com/big-data/hive-0-11-0-on-cloudera/



On Thu, Aug 22, 2013 at 8:29 AM, Jim Colestock <jc@ramblingredneck.com>wrote:

> Justin,
>
> That would be much appreciated!
>
> Thanks Again,
>
> JC
>
> On Aug 22, 2013, at 9:18 AM, Justin Workman <justinjworkman@gmail.com>
> wrote:
>
> I do have this working on CHD4.3.0 with MR1 and Kerberos.  I am finishing
> up a blog today explaining what was needed to get it working. I will reply
> with the link later today.
>
> Justin
>
> Sent from my iPhone
>
> On Aug 22, 2013, at 6:17 AM, Jim Colestock <jc@ramblingredneck.com> wrote:
>
> Hello All,
>
> Sorry for not replying to the thread, but I never received any of your
> responses..  Hopefully this info will help:
>
> @kulkarni.swarnim<http://search.gmane.org/?author=kulkarni.swarnim-Re5JQEeQqe8AvxtiuMwx3w%40public.gmane.org&sort=date>
>
> Here is the error I'm getting on task tracker for the mapper:
>
> 2013-08-21 15:34:32,491 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing
logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running child
> java.lang.RuntimeException: java.lang.NullPointerException
> 	at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
> 	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.lang.NullPointerException
> 	at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
> 	... 11 more
> 2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the
task
>
>
> @Srinivas Surasani<http://search.gmane.org/?author=Srinivas+Surasani&sort=date>
>  Hopefully the above helps..
>
> @Lars Francke <http://search.gmane.org/?author=Lars+Francke&sort=date>
>  Thanks, I'll take a peek at those 2 patches.. But I've just recently
> pulled down fresh code and recompiled everything and I'm still getting the
> above error.
>
> And for reference, below is my original email to the list..  Thanks all
> for you help!
>
>
> JC
>
>
>
> Hello All,
>
> Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3?
>
> I've been able to get hive to connect to my metadb (which is in Postgres).
>  Verified by doing a show tables..  I can run explain and describes on
> tables, but when I try to run anything that fires off an M/R job, I get the
> following error:
>
> hive>select count(*) from tableA;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
>   set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
>   set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
>   set mapred.reduce.tasks=<number>
> Starting Job = job_201307112247_13816, Tracking URL =
> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
> Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_201307112247_13816
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 1
> 2013-07-12 02:11:42,829 Stage-1 map = 0%,  reduce = 0%
> 2013-07-12 02:12:08,173 Stage-1 map = 100%,  reduce = 100%
> Ended Job = job_201307112247_13816 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL:
> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
> Examining task ID: task_201307112247_13816_m_000002 (and more) from job
> job_201307112247_13816
> Exception in thread "Thread-19" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/util/HostUtil
> at
> org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61)
> at
> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
> at
> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.util.HostUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> ... 4 more
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 0 msec
>
>
> I'm using my configs from hive 0.10, which works with no issues and this
> was pretty much a drop in replacement on the machine that hadoop 0.10 was
> running on..
>
> I've done a bunch of googling around and have found a bunch of other folks
> that have have had the same issue, but no solid answers..
>
> Thanks in advance for any help..
>
> JC
>
>
>
>

Mime
View raw message