spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wenc...@apache.org
Subject spark git commit: [SPARK-17424] Fix unsound substitution bug in ScalaReflection.
Date Fri, 12 May 2017 12:39:31 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.1 6e89d5740 -> 95de4672e


[SPARK-17424] Fix unsound substitution bug in ScalaReflection.

## What changes were proposed in this pull request?

This method gets a type's primary constructor and fills in type parameters with concrete types.
For example, `MapPartitions[T, U] -> MapPartitions[Int, String]`. This Substitution fails
when the actual type args are empty because they are still unknown. Instead, when there are
no resolved types to subsitute, this returns the original args with unresolved type parameters.
## How was this patch tested?

This doesn't affect substitutions where the type args are determined. This fixes our case
where the actual type args are empty and our job runs successfully.

Author: Ryan Blue <blue@apache.org>

Closes #15062 from rdblue/SPARK-17424-fix-unsound-reflect-substitution.

(cherry picked from commit b23693390781a99ff9248ea07a22e68884ffc747)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/95de4672
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/95de4672
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/95de4672

Branch: refs/heads/branch-2.1
Commit: 95de4672ea18e8cc0e14ee574386d252948e003d
Parents: 6e89d57
Author: Ryan Blue <blue@apache.org>
Authored: Fri May 12 20:38:36 2017 +0800
Committer: Wenchen Fan <wenchen@databricks.com>
Committed: Fri May 12 20:39:12 2017 +0800

----------------------------------------------------------------------
 .../org/apache/spark/sql/catalyst/ScalaReflection.scala | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/95de4672/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
----------------------------------------------------------------------
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
index fa1b900..1a1daae 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
@@ -814,8 +814,16 @@ trait ScalaReflection {
   def getConstructorParameters(tpe: Type): Seq[(String, Type)] = {
     val formalTypeArgs = tpe.typeSymbol.asClass.typeParams
     val TypeRef(_, _, actualTypeArgs) = tpe
-    constructParams(tpe).map { p =>
-      p.name.toString -> p.typeSignature.substituteTypes(formalTypeArgs, actualTypeArgs)
+    val params = constructParams(tpe)
+    // if there are type variables to fill in, do the substitution (SomeClass[T] -> SomeClass[Int])
+    if (actualTypeArgs.nonEmpty) {
+      params.map { p =>
+        p.name.toString -> p.typeSignature.substituteTypes(formalTypeArgs, actualTypeArgs)
+      }
+    } else {
+      params.map { p =>
+        p.name.toString -> p.typeSignature
+      }
     }
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message