= ;

On Aug 23, 2013, at 1:37 PM, = Justin Workman <justinjworkman@gmail.com> = wrote:

I just went through this same process = again and deployed to another 5 node cluster, and it is also working as = expected. I have ran anything very intensive yet, but did spawn several = mapreduce jobs.
On Fri, Aug 23, 2013 at 11:11 AM, Jim = Colestock <jc@ramblingredneck.com> = wrote:
Hello = Justin,

Thanks so much for the write up.. = I'm compiling now.. I did find one error in your = instructions:
patch -p0 > HIVE-4619.D10971.1.path
I believe it should = be:
patch -p0 < HIVE-4619.D10971.1.patch
But alas, I still = doesn't work for me. I managed to get it to compile after using = maven to install a zookeeper dependency.. But when I try to do anything = in hive that requires an M/R job, I get the following:

2013-08-23 13:07:07,865 INFO = org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater = with mapRetainSize=3D-1 and reduceRetainSize=3D-1
2013-08-23 = 13:07:07,867 WARN org.apache.hadoop.mapred.Child: Error running = child

java.lang.RuntimeException: = java.lang.NullPointerException
at = org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)=
at = org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255= )
at = org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(Hiv= eInputFormat.java:381)
at = org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(Hiv= eInputFormat.java:374)
at = org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(Combin= eHiveInputFormat.java:540)
at = org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
a= t org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at = org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at = java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1408)
at = org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: = java.lang.NullPointerException
at = org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)=
... 11 more
2013-08-23 13:07:07,871 INFO =
org.apache.hadoop.mapred.Task: Runnning cleanup for the =
task
Any thoughts?

Thanks = again,

JC

On Aug 22, 2013, at 11:15 = PM, Justin Workman <justinjworkman@gmail.com> = wrote:

For anyone interested, I have finished my blog on = getting Hive-0.11.0 working with CDH4.3.0. It is a hack and can be made = better but works like a charm.

http://www.justinjworkman.com/big-data/hive-0-11-0-on-cl= oudera/

On Thu, Aug 22, 2013 at 8:29 AM, Jim Colestock = <jc@ramblingredneck.com> wrote:

Justin,

That = would be much appreciated!

Thanks Again,

JC

=
On Aug 22, 2013, at 9:18 AM, Justin Workman <justinjworkman@gmail.com> wrote:

I do have this = working on CHD4.3.0 with MR1 and Kerberos. I am finishing up a = blog today explaining what was needed to get it working. I will reply = with the link later today.

Justin

Sent from my = iPhone

On Aug 22, 2013, at 6:17 AM, Jim Colestock <jc@ramblingredneck.com> wrote:

Hello All,

Sorry for not replying to the = thread, but I never received any of your responses.. Hopefully = this info will help:

@kulkarni.swarnim

Here is the error I'm getting on task tracker for = the mapper:

2013-08-21 15:34:32,491 INFO = org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater = with mapRetainSize=3D-1 and reduceRetainSize=3D-1 2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error = running child java.lang.RuntimeException: java.lang.NullPointerException at = org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)= at = org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255= ) at = org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(Hiv= eInputFormat.java:381) at = org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(Hiv= eInputFormat.java:374) at = org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(Combin= eHiveInputFormat.java:540) at = org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.NullPointerException at = org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)= ... 11 more 2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning = cleanup for the task

@Srinivas Surasani

Hopefully the = above helps..

@Lars= Francke

Thanks, I'll take = a peek at those 2 patches.. But I've just recently pulled down fresh = code and recompiled everything and I'm still getting the above = error.

And for reference, below is my original email to the = list.. Thanks all for you = help!

JC

Hello = All,

Has anyone = been successful at running hive 0.11 with Cloudera CDH 4.3? =

I've been able to get hive to connect to my metadb (which is in = Postgres). Verified by doing a show tables.. I can run = explain and describes on tables, but when I try to run anything that = fires off an M/R job, I get the following error:

hive>select count(*) from tableA;
Total = MapReduce jobs =3D 1
Launching Job 1 out of 1
Number = of reduce tasks determined at compile time: 1
In order to = change the average load for a reducer (in bytes):

set = hive.exec.reducers.bytes.per.reducer=3D<number>
In order = to limit the maximum number of reducers:
set = hive.exec.reducers.max=3D<number>
In order to set a = constant number of reducers:

set mapred.reduce.tasks=3D<number>
Starting = Job =3D job_201307112247_13816, Tracking URL =3D http://master:50030/jobdetails.jsp?jobid=3Djob_201307112= 247_13816

Kill Command =3D /usr/lib/hadoop/bin/hadoop job -kill = job_201307112247_13816
Hadoop job information for Stage-1: = number of mappers: 1; number of reducers: 1
2013-07-12 = 02:11:42,829 Stage-1 map =3D 0%, reduce =3D 0%

2013-07-12 02:12:08,173 Stage-1 map =3D 100%, reduce =3D = 100%
Ended Job =3D job_201307112247_13816 with = errors
Error during job, obtaining debugging = information...
Job Tracking URL: http://master:50030/jobdetails.jsp?jobid=3Djob_201307112= 247_13816

Examining task ID: task_201307112247_13816_m_000002 (and more) from = job job_201307112247_13816
Exception in thread "Thread-19" = java.lang.NoClassDefFoundError: = org/apache/hadoop/mapreduce/util/HostUtil

at = org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Sh= ims.java:61)
at = org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(Jo= bDebugger.java:186)

at = org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger= .java:142)
at = java.lang.Thread.run(Thread.java:619)

Caused by: java.lang.ClassNotFoundException: = org.apache.hadoop.mapreduce.util.HostUtil
at = java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)

at = java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at = java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at = sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)

at = java.lang.ClassLoader.loadClass(ClassLoader.java:248)
... 4 more
FAILED: Execution Error, return code 2 from = org.apache.hadoop.hive.ql.exec.MapRedTask

MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 = HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU = Time Spent: 0 msec

I'm using my configs from hive 0.10, which works with no issues and this = was pretty much a drop in replacement on the machine that hadoop 0.10 = was running on..

I've done = a bunch of googling around and have found a bunch of other folks that = have have had the same issue, but no solid answers..

Thanks in advance for any help..

JC