spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dav...@apache.org
Subject spark git commit: [SPARK-10459] [SQL] Do not need to have ConvertToSafe for PythonUDF
Date Thu, 17 Sep 2015 16:21:27 GMT
Repository: spark
Updated Branches:
  refs/heads/master e51345e1e -> 2a508df20


[SPARK-10459] [SQL] Do not need to have ConvertToSafe for PythonUDF

JIRA: https://issues.apache.org/jira/browse/SPARK-10459

As mentioned in the JIRA, `PythonUDF` actually could process `UnsafeRow`.

Specially, the rows in `childResults` in `BatchPythonEvaluation` will be projected to a `MutableRow`.
So I think we can enable `canProcessUnsafeRows` for `BatchPythonEvaluation` and get rid of
redundant `ConvertToSafe`.

Author: Liang-Chi Hsieh <viirya@appier.com>

Closes #8616 from viirya/pyudf-unsafe.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2a508df2
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2a508df2
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2a508df2

Branch: refs/heads/master
Commit: 2a508df20d03b3d4a3c05b65fb02d849bc080ef9
Parents: e51345e
Author: Liang-Chi Hsieh <viirya@appier.com>
Authored: Thu Sep 17 09:21:21 2015 -0700
Committer: Davies Liu <davies.liu@gmail.com>
Committed: Thu Sep 17 09:21:21 2015 -0700

----------------------------------------------------------------------
 .../main/scala/org/apache/spark/sql/execution/pythonUDFs.scala   | 4 ++++
 1 file changed, 4 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/2a508df2/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala
index 5a58d84..d0411da 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala
@@ -337,6 +337,10 @@ case class BatchPythonEvaluation(udf: PythonUDF, output: Seq[Attribute],
child:
 
   def children: Seq[SparkPlan] = child :: Nil
 
+  override def outputsUnsafeRows: Boolean = false
+  override def canProcessUnsafeRows: Boolean = true
+  override def canProcessSafeRows: Boolean = true
+
   protected override def doExecute(): RDD[InternalRow] = {
     val childResults = child.execute().map(_.copy())
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message