spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Balagopal Nair (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-10644) Applications wait even if free executors are available
Date Mon, 28 Sep 2015 15:50:05 GMT

    [ https://issues.apache.org/jira/browse/SPARK-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933475#comment-14933475
] 

Balagopal Nair commented on SPARK-10644:
----------------------------------------

I set both executor and worker memory to 512m.
Let me rephrase what I said before. Under full load, each host had about 1g RAM free. As I
said before, if this was a memory issue, spark will still try to launch jobs and the worker
processes would die. 



> Applications wait even if free executors are available
> ------------------------------------------------------
>
>                 Key: SPARK-10644
>                 URL: https://issues.apache.org/jira/browse/SPARK-10644
>             Project: Spark
>          Issue Type: Bug
>          Components: Scheduler
>    Affects Versions: 1.5.0
>         Environment: RHEL 6.5 64 bit
>            Reporter: Balagopal Nair
>            Priority: Minor
>
> Number of workers: 21
> Number of executors: 63
> Steps to reproduce:
> 1. Run 4 jobs each with max cores set to 10
> 2. The first 3 jobs run with 10 each. (30 executors consumed so far)
> 3. The 4 th job waits even though there are 33 idle executors.
> The reason is that a job will not get executors unless 
> the total number of EXECUTORS in use < the number of WORKERS
> If there are executors available, resources should be allocated to the pending job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message