spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From praveen seluka <praveen.sel...@gmail.com>
Subject Re: executorAdded event to DAGScheduler
Date Fri, 26 Sep 2014 12:26:01 GMT
In Yarn, we can easily  have multiple containers allocated in the same node.

On Fri, Sep 26, 2014 at 6:05 PM, Nan Zhu <zhunanmcgill@gmail.com> wrote:

>  just a quick reply, we cannot start two executors in the same host for a
> single application in the standard deployment (one worker per machine)
>
> I’m not sure if it will create an issue when you have multiple workers in
> the same host, as submitWaitingStages is called everywhere and I never
> try such a deployment mode
>
> Best,
>
> --
> Nan Zhu
>
> On Friday, September 26, 2014 at 8:02 AM, praveen seluka wrote:
>
> Can someone explain the motivation behind passing executorAdded event to
> DAGScheduler ? *DAGScheduler *does *submitWaitingStages *when *executorAdded
> *method is called by *TaskSchedulerImpl*. I see some issue in the below
> code,
>
> *TaskSchedulerImpl.scala code*
> if (!executorsByHost.contains(o.host)) {
>         executorsByHost(o.host) = new HashSet[String]()
>         executorAdded(o.executorId, o.host)
>         newExecAvail = true
>       }
>
> Note that executorAdded is called only when there is a new host and not
> for every new executor. For instance, there can be two executors in the
> same host and in this case. (But DAGScheduler executorAdded is notified
> only for new host - so only once in this case). If this is indeed an issue,
> I would like to submit a patch for this quickly. [cc Andrew Or]
>
> - Praveen
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message