spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Or (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-10790) Dynamic Allocation does not request any executors if first stage needs less than or equal to spark.dynamicAllocation.initialExecutors
Date Mon, 09 Nov 2015 18:45:11 GMT

     [ https://issues.apache.org/jira/browse/SPARK-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Or updated SPARK-10790:
------------------------------
    Affects Version/s:     (was: 1.5.0)
                       1.5.1

> Dynamic Allocation does not request any executors if first stage needs less than or equal
to spark.dynamicAllocation.initialExecutors
> -------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-10790
>                 URL: https://issues.apache.org/jira/browse/SPARK-10790
>             Project: Spark
>          Issue Type: Bug
>          Components: Scheduler
>    Affects Versions: 1.5.0
>            Reporter: Jonathan Kelly
>            Assignee: Saisai Shao
>             Fix For: 1.5.2, 1.6.0
>
>
> If you set spark.dynamicAllocation.initialExecutors > 0 (or spark.dynamicAllocation.minExecutors,
since spark.dynamicAllocation.initialExecutors defaults to spark.dynamicAllocation.minExecutors),
and the number of tasks in the first stage of your job is less than or equal to this min/init
number of executors, dynamic allocation won't actually request any executors and will just
hang indefinitely with the warning "Initial job has not accepted any resources; check your
cluster UI to ensure that workers are registered and have sufficient resources".
> The cause appears to be that ExecutorAllocationManager does not request any executors
while the application is still initializing, but it still sets the initial value of numExecutorsTarget
to spark.dynamicAllocation.initialExecutors. Once the job is running and has submitted its
first task, if the first task does not need more than spark.dynamicAllocation.initialExecutors,
ExecutorAllocationManager.updateAndSyncNumExecutorsTarget() does not think that it needs to
request any executors, so it doesn't.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message