hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stanley Shi <s...@gopivotal.com>
Subject Re: MR2 Job over LZO data
Date Fri, 07 Mar 2014 02:42:12 GMT
May be you can try download the LZO class and rebuild it against Hadoop
2.2.0;
If build success, you should be good to go;
if failed, then maybe you need to wait for the LZO guys to update their
code.

Regards,
*Stanley Shi,*



On Thu, Mar 6, 2014 at 6:29 PM, KingDavies <kingdavies@gmail.com> wrote:

> Running on Hadoop 2.2.0
>
> The Java MR2 job works as expected on an uncompressed data source using
> the TextInputFormat.class.
> But when using the LZO format the job fails:
> import com.hadoop.mapreduce.LzoTextInputFormat;
> job.setInputFormatClass(LzoTextInputFormat.class);
>
> Dependencies from the maven repository:
> http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.19/
> Also tried with elephant-bird-core 4.4
>
> The same data can be queried fine from within Hive(0.12) on the same
> cluster.
>
>
> The exception:
> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
> interface org.apache.hadoop.mapreduce.JobContext, but class was expected
>  at
> com.hadoop.mapreduce.LzoTextInputFormat.listStatus(LzoTextInputFormat.java:62)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:340)
>  at
> com.hadoop.mapreduce.LzoTextInputFormat.getSplits(LzoTextInputFormat.java:101)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:491)
>  at
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:508)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
>  at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>  at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
>  at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>  at com.cloudreach.DataQuality.Main.main(Main.java:42)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> I believe the issue is related to the changes in Hadoop 2, but where can I
> find a H2 compatible version?
>
> Thanks
>

Mime
View raw message