Return-Path: 
 <dev-return-10215-apmail-avro-dev-archive=avro.apache.org@avro.apache.org>
X-Original-To: apmail-avro-dev-archive@www.apache.org
Delivered-To: apmail-avro-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 033F4D44A
	for <apmail-avro-dev-archive@www.apache.org>;
 Thu, 20 Sep 2012 14:59:09 +0000 (UTC)
Received: (qmail 59927 invoked by uid 500); 20 Sep 2012 14:59:08 -0000
Delivered-To: apmail-avro-dev-archive@avro.apache.org
Received: (qmail 59577 invoked by uid 500); 20 Sep 2012 14:59:08 -0000
Mailing-List: contact dev-help@avro.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@avro.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@avro.apache.org>
List-Post: <mailto:dev@avro.apache.org>
List-Id: <dev.avro.apache.org>
Reply-To: dev@avro.apache.org
Delivered-To: mailing list dev@avro.apache.org
Received: (qmail 59524 invoked by uid 99); 20 Sep 2012 14:59:07 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Sep 2012 14:59:07 +0000
Date: Fri, 21 Sep 2012 01:59:07 +1100 (NCT)
From: "Tom White (JIRA)" <jira@apache.org>
To: dev@avro.apache.org
Message-ID: <792476805.103130.1348153147637.JavaMail.jiratomcat@arcas>
In-Reply-To: <441506003.97510.1348065367481.JavaMail.jiratomcat@arcas>
Subject: [jira] [Updated] (AVRO-1170) Avro's new mapreduce APIs don't work
 with Hadoop 2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/AVRO-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated AVRO-1170:
----------------------------

    Attachment: AVRO-1170.patch

Here's a new patch with the Maven changes.

For testing you can run {{mvn test -Dhadoop.version=2}} to test the mapred module with Hadoop 2. If you don't specify the {{hadoop.version}} property it defaults to 1 like the current behaviour. The other modules (i.e. not mapred) all build against Hadoop 1 since they use APIs that are binary compatible.

For building, the idea is to create a mapred jar with a hadoop1 or hadoop2 classifier. We also create a Hadoop 1 artifact with no classifier which is the default (for backwards compatibility). For this to work we build against Hadoop 2 first, then Hadoop 1 so that the JAR with no classifier is the last one build (Hadoop 1). I've changed the top-level build script to implement this.

For deployment, the instructions at https://cwiki.apache.org/confluence/display/AVRO/How+To+Release would need to change to deploy the JARs with classifiers. Locally deploying twice worked, although I'm not sure if this would work with a repository manager like Nexus:
{noformat}
mvn deploy -DskipTests=true -Dhadoop.version=2 -DaltDeploymentRepository=mine::default::file:///tmp/myrepo
mvn deploy -DskipTests=true -DaltDeploymentRepository=mine::default::file:///tmp/myrepo
{noformat}

For consumers of the Maven artifacts, if you didn't specify a classifier in your dependency section then it would use Hadoop 1, as before:

{noformat}
<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-mapred</artifactId>
  <version>1.7.2</version>
</dependency>
{noformat}

To use Hadoop 2, you would specify a hadoop2 classifier:

{noformat}
<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-mapred</artifactId>
  <version>1.7.2</version>
  <classifier>hadoop2</classifier>
</dependency>
{noformat}

                
> Avro's new mapreduce APIs don't work with Hadoop 2
> --------------------------------------------------
>
>                 Key: AVRO-1170
>                 URL: https://issues.apache.org/jira/browse/AVRO-1170
>             Project: Avro
>          Issue Type: Bug
>          Components: java
>    Affects Versions: 1.7.1
>            Reporter: Tom White
>         Attachments: AVRO-1170.patch, AVRO-1170.patch, AVRO-1170.patch
>
>
> Avro does not compile against Hadoop since some interfaces were changed to classes between Hadoop 1 and 2 (e.g. TaskAttemptContext).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira