spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mkhaitman <mark.khait...@chango.com>
Subject Re: Spark.Executor.Cores question
Date Tue, 27 Oct 2015 20:57:18 GMT
Hi Richard,

Thanks for the response. 

I should have added that the specific case where this becomes a problem is
when one of the executors for that application is lost/killed prematurely,
and the application attempts to spawn up a new executor without
consideration as to whether an executor already exists on the other node. 

In your example, if one of the executors dies for some reason (memory
exhaustion, or something crashed it), if there are still free cores on the
other nodes, it will spawn an extra executor, which can lead to further
memory problems on the other node that it just spawned on. 

Hopefully that clears up what I mean :) 

Mark.



--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-Executor-Cores-question-tp14763p14805.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message