spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lix...@apache.org
Subject spark git commit: [SPARK-22412][SQL] Fix incorrect comment in DataSourceScanExec
Date Sat, 04 Nov 2017 18:09:51 GMT
Repository: spark
Updated Branches:
  refs/heads/master 0c2aee69b -> f7f4e9c2d


[SPARK-22412][SQL] Fix incorrect comment in DataSourceScanExec

## What changes were proposed in this pull request?

Next fit decreasing bin packing algorithm is used to combine splits in DataSourceScanExec
but the comment incorrectly states that first fit decreasing algorithm is used. The current
implementation doesn't go back to a previously used bin other than the bin that the last element
was put into.

Author: Vinitha Gankidi <vgankidi@netflix.com>

Closes #19634 from vgankidi/SPARK-22412.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f7f4e9c2
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f7f4e9c2
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f7f4e9c2

Branch: refs/heads/master
Commit: f7f4e9c2db405b887832fcb592cd4522795d00ca
Parents: 0c2aee6
Author: Vinitha Gankidi <vgankidi@netflix.com>
Authored: Sat Nov 4 11:09:47 2017 -0700
Committer: gatorsmile <gatorsmile@gmail.com>
Committed: Sat Nov 4 11:09:47 2017 -0700

----------------------------------------------------------------------
 .../scala/org/apache/spark/sql/execution/DataSourceScanExec.scala  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/f7f4e9c2/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
index e9f6503..a607ec0 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
@@ -469,7 +469,7 @@ case class FileSourceScanExec(
       currentSize = 0
     }
 
-    // Assign files to partitions using "First Fit Decreasing" (FFD)
+    // Assign files to partitions using "Next Fit Decreasing"
     splitFiles.foreach { file =>
       if (currentSize + file.length > maxSplitBytes) {
         closePartition()


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message