avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AVRO-1170) Avro's new mapreduce APIs don't work with Hadoop 2
Date Thu, 20 Sep 2012 14:59:07 GMT

     [ https://issues.apache.org/jira/browse/AVRO-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tom White updated AVRO-1170:
----------------------------

    Attachment: AVRO-1170.patch

Here's a new patch with the Maven changes.

For testing you can run {{mvn test -Dhadoop.version=2}} to test the mapred module with Hadoop
2. If you don't specify the {{hadoop.version}} property it defaults to 1 like the current
behaviour. The other modules (i.e. not mapred) all build against Hadoop 1 since they use APIs
that are binary compatible.

For building, the idea is to create a mapred jar with a hadoop1 or hadoop2 classifier. We
also create a Hadoop 1 artifact with no classifier which is the default (for backwards compatibility).
For this to work we build against Hadoop 2 first, then Hadoop 1 so that the JAR with no classifier
is the last one build (Hadoop 1). I've changed the top-level build script to implement this.

For deployment, the instructions at https://cwiki.apache.org/confluence/display/AVRO/How+To+Release
would need to change to deploy the JARs with classifiers. Locally deploying twice worked,
although I'm not sure if this would work with a repository manager like Nexus:
{noformat}
mvn deploy -DskipTests=true -Dhadoop.version=2 -DaltDeploymentRepository=mine::default::file:///tmp/myrepo
mvn deploy -DskipTests=true -DaltDeploymentRepository=mine::default::file:///tmp/myrepo
{noformat}

For consumers of the Maven artifacts, if you didn't specify a classifier in your dependency
section then it would use Hadoop 1, as before:

{noformat}
<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-mapred</artifactId>
  <version>1.7.2</version>
</dependency>
{noformat}

To use Hadoop 2, you would specify a hadoop2 classifier:

{noformat}
<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-mapred</artifactId>
  <version>1.7.2</version>
  <classifier>hadoop2</classifier>
</dependency>
{noformat}

                
> Avro's new mapreduce APIs don't work with Hadoop 2
> --------------------------------------------------
>
>                 Key: AVRO-1170
>                 URL: https://issues.apache.org/jira/browse/AVRO-1170
>             Project: Avro
>          Issue Type: Bug
>          Components: java
>    Affects Versions: 1.7.1
>            Reporter: Tom White
>         Attachments: AVRO-1170.patch, AVRO-1170.patch, AVRO-1170.patch
>
>
> Avro does not compile against Hadoop since some interfaces were changed to classes between
Hadoop 1 and 2 (e.g. TaskAttemptContext).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message