spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Fwd: [jira] [Created] (HADOOP-12527) Upgrade Avro dependency to 1.7.7
Date Thu, 29 Oct 2015 10:42:20 GMT
Coming from AWS dev team.

he's probably got a point: even if you cut avro from the hadoop POM transitives, if you run
in a Hadoop cluster with the hadoop classpaths getting in, there could be conflict.

1. has anyone seen this?
2. when spark when up to avro 1.7.7 —what broke? It's that risk of breaking downstream things
that leads to hadoop JARs being frozen


Begin forwarded message:

From: "Jonathan Kelly (JIRA)" <jira@apache.org<mailto:jira@apache.org>>
Date: 29 October 2015 at 00:52:27 GMT
To: <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org>>
Subject: [jira] [Created] (HADOOP-12527) Upgrade Avro dependency to 1.7.7

Jonathan Kelly created HADOOP-12527:
---------------------------------------

            Summary: Upgrade Avro dependency to 1.7.7
                Key: HADOOP-12527
                URL: https://issues.apache.org/jira/browse/HADOOP-12527
            Project: Hadoop Common
         Issue Type: Improvement
   Affects Versions: 2.7.1
           Reporter: Jonathan Kelly


Hadoop has depended upon Avro 1.7.4 for a couple of years now (see HADOOP-9672), but Apache
Spark depends upon what is currently the latest version of Avro (1.7.7).

This can cause issues if Spark is configured to include the full Hadoop classpath, as the
classpath would then contain both Avro 1.7.4 and 1.7.7, with the 1.7.4 classes possibly winning
depending on ordering. Here is an example of this issue: http://stackoverflow.com/questions/33159254/avro-error-on-aws-emr/33403111#33403111

Would it be possible to upgrade Hadoop's Avro dependency to 1.7.7 now?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Mime
View raw message