spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lix...@apache.org
Subject spark git commit: [SPARK-24768][FOLLOWUP][SQL] Avro migration followup: change artifactId to spark-avro
Date Wed, 25 Jul 2018 15:42:49 GMT
Repository: spark
Updated Branches:
  refs/heads/master 7a5fd4a91 -> c44eb561e


[SPARK-24768][FOLLOWUP][SQL] Avro migration followup: change artifactId to spark-avro

## What changes were proposed in this pull request?
After rethinking on the artifactId, I think it should be `spark-avro` instead of `spark-sql-avro`,
which is simpler, and consistent with the previous artifactId. I think we need to change it
before Spark 2.4 release.

Also a tiny change: use `spark.sessionState.newHadoopConf()` to get the hadoop configuration,
thus the related hadoop configurations in SQLConf will come into effect.

## How was this patch tested?

Unit test

Author: Gengliang Wang <gengliang.wang@databricks.com>

Closes #21866 from gengliangwang/avro_followup.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c44eb561
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c44eb561
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c44eb561

Branch: refs/heads/master
Commit: c44eb561ec371af0405710d2e9358f9797655145
Parents: 7a5fd4a
Author: Gengliang Wang <gengliang.wang@databricks.com>
Authored: Wed Jul 25 08:42:45 2018 -0700
Committer: Xiao Li <gatorsmile@gmail.com>
Committed: Wed Jul 25 08:42:45 2018 -0700

----------------------------------------------------------------------
 external/avro/pom.xml                                              | 2 +-
 .../src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/c44eb561/external/avro/pom.xml
----------------------------------------------------------------------
diff --git a/external/avro/pom.xml b/external/avro/pom.xml
index ad7df1f..8f118ba 100644
--- a/external/avro/pom.xml
+++ b/external/avro/pom.xml
@@ -25,7 +25,7 @@
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-sql-avro_2.11</artifactId>
+  <artifactId>spark-avro_2.11</artifactId>
   <properties>
     <sbt.project.name>avro</sbt.project.name>
   </properties>

http://git-wip-us.apache.org/repos/asf/spark/blob/c44eb561/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala
----------------------------------------------------------------------
diff --git a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala
index b043252..c6b3c13 100755
--- a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala
+++ b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala
@@ -56,7 +56,7 @@ private[avro] class AvroFileFormat extends FileFormat with DataSourceRegister
{
       spark: SparkSession,
       options: Map[String, String],
       files: Seq[FileStatus]): Option[StructType] = {
-    val conf = spark.sparkContext.hadoopConfiguration
+    val conf = spark.sessionState.newHadoopConf()
     val parsedOptions = new AvroOptions(options, conf)
 
     // Schema evolution is not supported yet. Here we only pick a single random sample file
to


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message