spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From samee...@apache.org
Subject spark git commit: [SPARK-23310][CORE] Turn off read ahead input stream for unshafe shuffle reader
Date Mon, 05 Feb 2018 18:20:06 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.3 e688ffee2 -> 173449c2b


[SPARK-23310][CORE] Turn off read ahead input stream for unshafe shuffle reader

To fix regression for TPC-DS queries

Author: Sital Kedia <skedia@fb.com>

Closes #20492 from sitalkedia/turn_off_async_inputstream.

(cherry picked from commit 03b7e120dd7ff7848c936c7a23644da5bd7219ab)
Signed-off-by: Sameer Agarwal <sameerag@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/173449c2
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/173449c2
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/173449c2

Branch: refs/heads/branch-2.3
Commit: 173449c2bd454a87487f8eebf7d74ee6ed505290
Parents: e688ffe
Author: Sital Kedia <skedia@fb.com>
Authored: Mon Feb 5 10:19:18 2018 -0800
Committer: Sameer Agarwal <sameerag@apache.org>
Committed: Mon Feb 5 10:20:02 2018 -0800

----------------------------------------------------------------------
 .../util/collection/unsafe/sort/UnsafeSorterSpillReader.java     | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/173449c2/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java
b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java
index e2f48e5..71e7c7a 100644
--- a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java
+++ b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java
@@ -76,8 +76,10 @@ public final class UnsafeSorterSpillReader extends UnsafeSorterIterator
implemen
         SparkEnv.get() == null ? 0.5 :
              SparkEnv.get().conf().getDouble("spark.unsafe.sorter.spill.read.ahead.fraction",
0.5);
 
+    // SPARK-23310: Disable read-ahead input stream, because it is causing lock contention
and perf regression for
+    // TPC-DS queries.
     final boolean readAheadEnabled = SparkEnv.get() != null &&
-        SparkEnv.get().conf().getBoolean("spark.unsafe.sorter.spill.read.ahead.enabled",
true);
+        SparkEnv.get().conf().getBoolean("spark.unsafe.sorter.spill.read.ahead.enabled",
false);
 
     final InputStream bs =
         new NioBufferedFileInputStream(file, (int) bufferSizeBytes);


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message