spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mukesh Jha <me.mukesh....@gmail.com>
Subject SPARK-streaming app running 10x slower on YARN vs STANDALONE cluster
Date Mon, 29 Dec 2014 12:36:23 GMT
Hello Experts,
I'm bench-marking Spark on YARN (
https://spark.apache.org/docs/latest/running-on-yarn.html) vs a standalone
spark cluster (https://spark.apache.org/docs/latest/spark-standalone.html).
I have a standalone cluster with 3 executors, and a spark app running on
yarn with 4 executors as shown below.

The spark job running inside yarn is 10x slower than the one running on the
standalone cluster (even though the yarn has more number of workers), also
in both the case all the executors are in the same datacenter so there
shouldn't be any latency. On YARN each 5sec batch is reading data from
kafka and processing it in 5sec & on the standalone cluster each 5sec batch
is getting processed in 0.4sec.
Also, In YARN mode all the executors are not getting used up evenly as
vm-13 & vm-14 are running most of the tasks whereas in the standalone mode
all the executors are running the tasks.

Do I need to set up some configuration to evenly distribute the tasks? Also
do you have any pointers on the reasons the yarn job is 10x slower than the
standalone job?
Any suggestion is greatly appreciated, Thanks in advance.

YARN(5 workers + driver)
========================
Executor ID Address RDD Blocks Memory Used DU  AT FT CT TT TT Input ShuffleRead
ShuffleWrite Thread Dump
1 vm-18.cloud.com:51796 0 0.0B/530.3MB 0.0 B 1 0 16 17 634 ms 0.0 B 2047.0
B 1710.0 B Thread Dump
2 vm-13.cloud.com:57264 0 0.0B/530.3MB 0.0 B 0 0 1427 1427 5.5 m 0.0 B 0.0
B 0.0 B Thread Dump
3 vm-14.cloud.com:54570 0 0.0B/530.3MB 0.0 B 0 0 1379 1379 5.2 m 0.0 B 1368.0
B 2.8 KB Thread Dump
4 vm-11.cloud.com:56201 0 0.0B/530.3MB 0.0 B 0 0 10 10 625 ms 0.0 B 1368.0
B 1026.0 B Thread Dump
5 vm-5.cloud.com:42958 0 0.0B/530.3MB 0.0 B 0 0 22 22 632 ms 0.0 B 1881.0 B 2.8
KB Thread Dump
<driver> vm.cloud.com:51847 0 0.0B/530.0MB 0.0 B 0 0 0 0 0 ms 0.0 B 0.0 B 0.0
B Thread Dump

/homext/spark/bin/spark-submit
--master yarn-cluster --num-executors 2 --driver-memory 512m
--executor-memory 512m --executor-cores 2
--class com.oracle.ci.CmsgK2H /homext/lib/MJ-ci-k2h.jar
vm.cloud.com:2181/kafka spark-yarn avro 1 5000

STANDALONE(3 workers + driver)
==============================
Executor ID Address RDD Blocks Memory Used DU AT FT CT TT TT Input ShuffleRead
ShuffleWrite Thread Dump
0 vm-71.cloud.com:55912 0 0.0B/265.0MB 0.0 B 0 0 1069 1069 6.0 m 0.0 B 1534.0
B 3.0 KB Thread Dump
1 vm-72.cloud.com:40897 0 0.0B/265.0MB 0.0 B 0 0 1057 1057 5.9 m 0.0 B 1368.0
B 4.0 KB Thread Dump
2 vm-73.cloud.com:37621 0 0.0B/265.0MB 0.0 B 1 0 1059 1060 5.9 m 0.0 B 2.0
KB 1368.0 B Thread Dump
<driver> vm.cloud.com:58299 0 0.0B/265.0MB 0.0 B 0 0 0 0 0 ms 0.0 B 0.0 B 0.0
B Thread Dump

/homext/spark/bin/spark-submit
--master spark://chsnmvproc71vm3.usdc2.oraclecloud.com:7077
--class com.oracle.ci.CmsgK2H /homext/lib/MJ-ci-k2h.jar
vm.cloud.com:2181/kafka spark-standalone avro 1 5000

PS: I did go through the spark website and
http://www.virdata.com/tuning-spark/, but was out of any luck.

-- 
Cheers,
Mukesh Jha

Mime
View raw message