hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Miller <Alan.Mil...@synopsys.com>
Subject Re: MR job runs on CDH3 but not on CDH4
Date Wed, 04 Jul 2012 19:32:46 GMT
Ok thanks, I'll post there.

I realized that the issue has to do with the extra jars that I added to the hadoop installation.

My job wasnt getting submitted becauae my tasktrackers don't seem to start if I have  avro-1.7.0.jar
& avro-tools-1.7.0.jar in my hadoop/lib directory.

But I need these jars in order to write the avro file.

Alan

Sent from my iPhone

On Jul 4, 2012, at 19:21, "Arun C Murthy" <acm@hortonworks.com<mailto:acm@hortonworks.com>>
wrote:

It's hard for folks here to help you on CDH - please ask their own user lists.

Arun

On Jul 4, 2012, at 8:49 AM, Alan Miller wrote:

Hi,

I’m trying to move from CDH3U3 to CDH4.
My existing MR program works fine on CDH3U3  but I cant get it to run on CDH4.

Basically my Driver class
1.       queries a PG DB and writes some HashMaps to files in the Distributed Cache,
2.       then writes some Avro files (avro 1.7.0) to HDFS,
3.       and then triggers a MRv1 job to process the Avro files.

The DC & Avro files get written so HDFS is working, but my job is not getting started.
I get an error:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy()
…
                at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478)

Seems the job doesn’t even get accepted .
At  MyDriver:397 (see below) I call job.submit, but that’s it.

…..
12/07/04 08:16:54 INFO MyDriver: Processing data [my-data]
12/07/04 08:16:54 INFO MyDriver: Write Avro: [Tue Jul 03 00:00:00 PDT 2012 > etime <=
Tue Jul 03 23:59:59 PDT 2012]
12/07/04 08:19:47 INFO MyDriver: Initialized file /data/in/my-data_2012-07-03.avro
12/07/04 08:19:49 INFO MyDriver: Read 158285 lines, Wrote 158131 records to 1 file(s)
12/07/04 08:19:49 INFO MyDriver: Wed Jul 04 08:19:49 PDT 2012 Finished avro data /data/in/my-data_2012-07-03.avro
12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/83206b5c-8a1c-46f3-bfb2-d8c3e949a530#q_map
to distributed cache.
12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/b2ebfeb9-bdb0-489e-8186-8e18f4416224#u_map
to distributed cache.
12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/437cfd91-aa07-4c3a-b4c9-cd4ae076f7ad#r_map
to distributed cache.
12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/9554fe48-2171-423c-ba54-6249ffc882d4#m_map
to distributed cache.
12/07/04 08:19:49 INFO MyDriver: Added /data/in/y-data_2012-07-03.avro to input files list.
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/security/UserGroupInformation;Lorg/apache/hadoop/conf/Configuration;Ljavax/net/SocketFactory;)Lorg/apache/hadoop/ipc/VersionedProtocol;
                at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478)
                at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472)
                at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:455)
                at org.apache.hadoop.mapreduce.Job$1.run(Job.java:478)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Subject.java:396)
                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
                at org.apache.hadoop.mapreduce.Job.connect(Job.java:476)
                at org.apache.hadoop.mapreduce.Job.submit(Job.java:464)
                at com.mycompany.MyDriver.run(MyDriver.java:397)
                at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
                at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
                at com.mycompany.MyDriver.runHadoopJob(MyDriver.java:308)
                at com.mycompany.MyDriver.main(MyDriver.java:1532)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.util.RunJar.main(RunJar.java:208)


Alan

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



Mime
View raw message