spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-20343][BUILD] Force Avro 1.7.7 in sbt build to resolve build failure in SBT Hadoop 2.6 master on Jenkins
Date Tue, 18 Apr 2017 10:05:06 GMT
Repository: spark
Updated Branches:
  refs/heads/master 07fd94e0d -> d4f10cbbe


[SPARK-20343][BUILD] Force Avro 1.7.7 in sbt build to resolve build failure in SBT Hadoop
2.6 master on Jenkins

## What changes were proposed in this pull request?

This PR proposes to force Avro's version to 1.7.7 in core to resolve the build failure as
below:

```
[error] /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123:
value createDatumWriter is not a member of org.apache.avro.generic.GenericData
[error]     writerCache.getOrElseUpdate(schema, GenericData.get.createDatumWriter(schema))
[error]
```

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/2770/consoleFull

Note that this is a hack and should be removed in the future.

## How was this patch tested?

I only tested this actually overrides the dependency.

I tried many ways but I was unable to reproduce this in my local. Sean also tried the way
I did but he was also unable to reproduce this.

Please refer the comments in https://github.com/apache/spark/pull/17477#issuecomment-294094092

Author: hyukjinkwon <gurwls223@gmail.com>

Closes #17651 from HyukjinKwon/SPARK-20343-sbt.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d4f10cbb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d4f10cbb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d4f10cbb

Branch: refs/heads/master
Commit: d4f10cbbe1b9d13e43d80a50d204781e1c5c2da9
Parents: 07fd94e
Author: hyukjinkwon <gurwls223@gmail.com>
Authored: Tue Apr 18 11:05:00 2017 +0100
Committer: Sean Owen <sowen@cloudera.com>
Committed: Tue Apr 18 11:05:00 2017 +0100

----------------------------------------------------------------------
 pom.xml                  |  1 +
 project/SparkBuild.scala | 14 ++++++++++++--
 2 files changed, 13 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/d4f10cbb/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index c117459..14370d9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -142,6 +142,7 @@
     <ivy.version>2.4.0</ivy.version>
     <oro.version>2.0.8</oro.version>
     <codahale.metrics.version>3.1.2</codahale.metrics.version>
+    <!-- Keep consistent with Avro vesion in SBT build for SPARK-20343 -->
     <avro.version>1.7.7</avro.version>
     <avro.mapred.classifier>hadoop2</avro.mapred.classifier>
     <jets3t.version>0.9.3</jets3t.version>

http://git-wip-us.apache.org/repos/asf/spark/blob/d4f10cbb/project/SparkBuild.scala
----------------------------------------------------------------------
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index e52baf5..77dae28 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -318,8 +318,8 @@ object SparkBuild extends PomBuild {
     enable(MimaBuild.mimaSettings(sparkHome, x))(x)
   }
 
-  /* Generate and pick the spark build info from extra-resources */
-  enable(Core.settings)(core)
+  /* Generate and pick the spark build info from extra-resources and override a dependency
*/
+  enable(Core.settings ++ CoreDependencyOverrides.settings)(core)
 
   /* Unsafe settings */
   enable(Unsafe.settings)(unsafe)
@@ -444,6 +444,16 @@ object DockerIntegrationTests {
 }
 
 /**
+ * Overrides to work around sbt's dependency resolution being different from Maven's in Unidoc.
+ *
+ * Note that, this is a hack that should be removed in the future. See SPARK-20343
+ */
+object CoreDependencyOverrides {
+  lazy val settings = Seq(
+    dependencyOverrides += "org.apache.avro" % "avro" % "1.7.7")
+}
+
+/**
  * Overrides to work around sbt's dependency resolution being different from Maven's.
  */
 object DependencyOverrides {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message