spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-22870][CORE] Dynamic allocation should allow 0 idle time
Date Sat, 13 Jan 2018 16:01:56 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.3 ca27d9cb5 -> 801ffd799


[SPARK-22870][CORE] Dynamic allocation should allow 0 idle time

## What changes were proposed in this pull request?

This pr to make `0` as a valid value for `spark.dynamicAllocation.executorIdleTimeout`.
For details, see the jira description: https://issues.apache.org/jira/browse/SPARK-22870.

## How was this patch tested?

N/A

Author: Yuming Wang <yumwang@ebay.com>
Author: Yuming Wang <wgyumg@gmail.com>

Closes #20080 from wangyum/SPARK-22870.

(cherry picked from commit fc6fe8a1d0f161c4788f3db94de49a8669ba3bcc)
Signed-off-by: Sean Owen <sowen@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/801ffd79
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/801ffd79
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/801ffd79

Branch: refs/heads/branch-2.3
Commit: 801ffd799922e1c2751d3331874b88a67da8cf01
Parents: ca27d9c
Author: Yuming Wang <yumwang@ebay.com>
Authored: Sat Jan 13 10:01:44 2018 -0600
Committer: Sean Owen <sowen@cloudera.com>
Committed: Sat Jan 13 10:01:53 2018 -0600

----------------------------------------------------------------------
 .../scala/org/apache/spark/ExecutorAllocationManager.scala    | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/801ffd79/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
index 2e00dc8..6c59038 100644
--- a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
+++ b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
@@ -195,8 +195,11 @@ private[spark] class ExecutorAllocationManager(
       throw new SparkException(
         "spark.dynamicAllocation.sustainedSchedulerBacklogTimeout must be > 0!")
     }
-    if (executorIdleTimeoutS <= 0) {
-      throw new SparkException("spark.dynamicAllocation.executorIdleTimeout must be >
0!")
+    if (executorIdleTimeoutS < 0) {
+      throw new SparkException("spark.dynamicAllocation.executorIdleTimeout must be >=
0!")
+    }
+    if (cachedExecutorIdleTimeoutS < 0) {
+      throw new SparkException("spark.dynamicAllocation.cachedExecutorIdleTimeout must be
>= 0!")
     }
     // Require external shuffle service for dynamic allocation
     // Otherwise, we may lose shuffle files when killing executors


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message