spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0
Date Fri, 03 Feb 2017 21:37:07 GMT
Hi,

Just to throw few zlotys to the conversation, I believe that Spark
Standalone does not enforce any memory checks to limit or even kill
executors beyond requested memory (like YARN). I also found that
memory does not have much of use while scheduling tasks and CPU
matters only.

My understanding of `spark.memory.offHeap.enabled` is `false` is that
it does not disable off heap memory used in Java NIO for buffers in
shuffling, RPC, etc. so the memory is always (?) more than you request
for mx using executor-memory.

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Sun, Jan 22, 2017 at 9:57 AM, StanZhai <mail@zhaishidan.cn> wrote:
> Hi all,
>
> We just upgraded our Spark from 1.6.2 to 2.1.0.
>
> Our Spark application is started by spark-submit with config of
> `--executor-memory 35G` in standalone model, but the actual use of memory up
> to 65G after a full gc(jmap -histo:live $pid) as follow:
>
> test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend
> test      181941 181 34.7 94665384 68836752 ?   Sl   09:25 711:21
> /home/test/service/jdk/bin/java -cp
> /home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/
> -Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc
> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
> spark://CoarseGrainedScheduler@xxx.xxx.xxx.xxx:47781 --executor-id 1
> --hostname test-192 --cores 36 --app-id app-20170122092509-0017 --worker-url
> spark://Worker@test-192:33890
>
> Our Spark jobs are all sql.
>
> The exceed memory looks like off-heap memory, but the default value of
> `spark.memory.offHeap.enabled` is `false`.
>
> We didn't find the problem in Spark 1.6.x, what causes this in Spark 2.1.0?
>
> Any help is greatly appreicated!
>
> Best,
> Stan
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message