cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Coverston (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
Date Tue, 03 Dec 2013 22:59:36 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Benjamin Coverston updated CASSANDRA-5201:
------------------------------------------

    Attachment: hadoopCompat.patch

Poking around at other projects this generally gets solved in one of two ways: Ship two versions
of their Hadoop integrations (one compiled for the old, and one compiled for the new), or
use a little reflection to make things work across the board.

I'm attaching a patch that uses the hadoopCompat subproject of elephantbird. This will allow
us to compile a single binary and run with the new and old context objects.

I've tested this patch with HDP 2.0, and Apache Hadoop 1.0.4 and it works fine with both (including
Hive in DSE). With Pig I needed to compile our (optional) pig dependency with:

bq. ant clean jar-withouthadoop -Dhadoopversion=23

Only really needed if you're using one of the current versions of thrift with the new JobContext.


> Cassandra/Hadoop does not support current Hadoop releases
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-5201
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5201
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Hadoop
>    Affects Versions: 1.2.0
>            Reporter: Brian Jeltema
>            Assignee: Dave Brosius
>         Attachments: 5201_a.txt, hadoopCompat.patch
>
>
> Using Hadoop 0.22.0 with Cassandra results in the stack trace below.
> It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext
> from a class to an interface.
> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext,
but class was expected
> 	at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357)
> 	at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045)
> 	at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
> 	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042)
> 	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062)
> 	at MyHadoopApp.run(MyHadoopApp.java:163)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> 	at MyHadoopApp.main(MyHadoopApp.java:82)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:192)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message