hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Justin Workman <justinjwork...@gmail.com>
Subject Re: Hive 0.11 with Cloudera CHD4.3 MR v1
Date Thu, 22 Aug 2013 13:18:39 GMT
I do have this working on CHD4.3.0 with MR1 and Kerberos.  I am finishing
up a blog today explaining what was needed to get it working. I will reply
with the link later today.

Justin

Sent from my iPhone

On Aug 22, 2013, at 6:17 AM, Jim Colestock <jc@ramblingredneck.com> wrote:

Hello All,

Sorry for not replying to the thread, but I never received any of your
responses..  Hopefully this info will help:

@kulkarni.swarnim<http://search.gmane.org/?author=kulkarni.swarnim-Re5JQEeQqe8AvxtiuMwx3w%40public.gmane.org&sort=date>

Here is the error I'm getting on task tracker for the mapper:

2013-08-21 15:34:32,491 INFO
org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs'
truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: java.lang.NullPointerException
	at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
	... 11 more
2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning
cleanup for the task


@Srinivas Surasani<http://search.gmane.org/?author=Srinivas+Surasani&sort=date>
Hopefully the above helps..

@Lars Francke <http://search.gmane.org/?author=Lars+Francke&sort=date>
Thanks, I'll take a peek at those 2 patches.. But I've just recently pulled
down fresh code and recompiled everything and I'm still getting the above
error.

And for reference, below is my original email to the list..  Thanks all for
you help!


JC



Hello All,

Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3?

I've been able to get hive to connect to my metadb (which is in Postgres).
 Verified by doing a show tables..  I can run explain and describes on
tables, but when I try to run anything that fires off an M/R job, I get the
following error:

hive>select count(*) from tableA;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_201307112247_13816, Tracking URL =
http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_201307112247_13816
Hadoop job information for Stage-1: number of mappers: 1; number of
reducers: 1
2013-07-12 02:11:42,829 Stage-1 map = 0%,  reduce = 0%
2013-07-12 02:12:08,173 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201307112247_13816 with errors
Error during job, obtaining debugging information...
Job Tracking URL:
http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
Examining task ID: task_201307112247_13816_m_000002 (and more) from job
job_201307112247_13816
Exception in thread "Thread-19" java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/util/HostUtil
at
org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61)
at
org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
at
org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapreduce.util.HostUtil
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
... 4 more
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec


I'm using my configs from hive 0.10, which works with no issues and this
was pretty much a drop in replacement on the machine that hadoop 0.10 was
running on..

I've done a bunch of googling around and have found a bunch of other folks
that have have had the same issue, but no solid answers..

Thanks in advance for any help..

JC

Mime
View raw message