spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From l...@apache.org
Subject spark git commit: [SPARK-8841] [SQL] Fix partition pruning percentage log message
Date Mon, 06 Jul 2015 17:17:28 GMT
Repository: spark
Updated Branches:
  refs/heads/master 86768b7b3 -> 39e4e7e4d


[SPARK-8841] [SQL] Fix partition pruning percentage log message

When pruning partitions for a query plan, a message is logged indicating what how many partitions
were selected based on predicate criteria, and what percent were pruned.

The current release erroneously uses `1 - total/selected` to compute this quantity, leading
to nonsense messages like "pruned -1000% partitions". The fix is simple and obvious.

Author: Steve Lindemann <steve.lindemann@engineersgatelp.com>

Closes #7227 from srlindemann/master and squashes the following commits:

c788061 [Steve Lindemann] fix percentPruned log message


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/39e4e7e4
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/39e4e7e4
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/39e4e7e4

Branch: refs/heads/master
Commit: 39e4e7e4d89077a637c4cad3a986e0e3447d1ae7
Parents: 86768b7
Author: Steve Lindemann <steve.lindemann@engineersgatelp.com>
Authored: Mon Jul 6 10:17:05 2015 -0700
Committer: Cheng Lian <lian@databricks.com>
Committed: Mon Jul 6 10:17:05 2015 -0700

----------------------------------------------------------------------
 .../scala/org/apache/spark/sql/sources/DataSourceStrategy.scala    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/39e4e7e4/sql/core/src/main/scala/org/apache/spark/sql/sources/DataSourceStrategy.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/sources/DataSourceStrategy.scala
b/sql/core/src/main/scala/org/apache/spark/sql/sources/DataSourceStrategy.scala
index ce16e05..66f7ba9 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/sources/DataSourceStrategy.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/sources/DataSourceStrategy.scala
@@ -65,7 +65,7 @@ private[sql] object DataSourceStrategy extends Strategy with Logging {
       logInfo {
         val total = t.partitionSpec.partitions.length
         val selected = selectedPartitions.length
-        val percentPruned = (1 - total.toDouble / selected.toDouble) * 100
+        val percentPruned = (1 - selected.toDouble / total.toDouble) * 100
         s"Selected $selected partitions out of $total, pruned $percentPruned% partitions."
       }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message