spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-21746) nondeterministic expressions incorrectly for filter predicates
Date Mon, 28 Aug 2017 11:13:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143655#comment-16143655
] 

Apache Spark commented on SPARK-21746:
--------------------------------------

User 'heary-cao' has created a pull request for this issue:
https://github.com/apache/spark/pull/18961

> nondeterministic expressions incorrectly for filter predicates
> --------------------------------------------------------------
>
>                 Key: SPARK-21746
>                 URL: https://issues.apache.org/jira/browse/SPARK-21746
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: caoxuewen
>
> Currently, We do interpretedpredicate optimization, but not very well, because when our
filter contained an indeterminate expression, it would have an exception. This PR describes
solving this problem by adding the initialize method in InterpretedPredicate.
> java.lang.IllegalArgumentException:
> java.lang.IllegalArgumentException: requirement failed: Nondeterministic expression org.apache.spark.sql.catalyst.expressions.Rand
should be initialized before eval.
> 	at scala.Predef$.require(Predef.scala:224)
> 	at org.apache.spark.sql.catalyst.expressions.Nondeterministic$class.eval(Expression.scala:291)
> 	at org.apache.spark.sql.catalyst.expressions.RDG.eval(randomExpressions.scala:34)
> 	at org.apache.spark.sql.catalyst.expressions.BinaryExpression.eval(Expression.scala:415)
> 	at org.apache.spark.sql.catalyst.expressions.InterpretedPredicate.eval(predicates.scala:38)
> 	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils$$anonfun$prunePartitionsByFilter$1.apply(ExternalCatalogUtils.scala:158)
> 	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils$$anonfun$prunePartitionsByFilter$1.apply(ExternalCatalogUtils.scala:157)
> 	at scala.collection.immutable.Stream.filter(Stream.scala:519)
> 	at scala.collection.immutable.Stream.filter(Stream.scala:202)
> 	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils$.prunePartitionsByFilter(ExternalCatalogUtils.scala:157)
> 	at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$listPartitionsByFilter$1.apply(HiveExternalCatalog.scala:1129)
> 	at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$listPartitionsByFilter$1.apply(HiveExternalCatalog.scala:1119)
> 	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
> 	at org.apache.spark.sql.hive.HiveExternalCatalog.listPartitionsByFilter(HiveExternalCatalog.scala:1119)
> 	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listPartitionsByFilter(SessionCatalog.scala:925)
> 	at org.apache.spark.sql.execution.datasources.CatalogFileIndex.filterPartitions(CatalogFileIndex.scala:73)
> 	at org.apache.spark.sql.execution.datasources.PruneFileSourcePartitions$$anonfun$apply$1.applyOrElse(PruneFileSourcePartitions.scala:60)
> 	at org.apache.spark.sql.execution.datasources.PruneFileSourcePartitions$$anonfun$apply$1.applyOrElse(PruneFileSourcePartitions.scala:27)
> 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
> 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
> 	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
> 	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266)
> 	at org.apache.spark.sql.execution.datasources.PruneFileSourcePartitions$.apply(PruneFileSourcePartitions.scala:27)
> 	at org.apache.spark.sql.execution.datasources.PruneFileSourcePartitions$.apply(PruneFileSourcePartitions.scala:26)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message