spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Punit Naik <naik.puni...@gmail.com>
Subject Executor Lost error
Date Tue, 04 Oct 2016 00:07:22 GMT
Hi All

I am trying to run a program for a large dataset (~ 1TB). I have already
tested the code for low size of data and it works fine. But what I noticed
is that he job fails if the size of input is large. It was giving me errors
regarding akkka actor disassociation which I fixed by increasing the
timeouts.
But now I am getting errors like "execuyor lost" and "executor lost
failure" which I can't seem to figure out. These are my current set of
configs:

--conf "spark.network.timeout=30000"
--conf "spark.core.connection.ack.wait.timeout=30000"
--conf "spark.akka.timeout=30000"
--conf "spark.akka.askTimeout=30000"
--conf "spark.akka.frameSize=1000"
--conf "spark.storage.blockManagerSlaveTimeoutMs=600000"
--conf "spark.network.timeout=600"
--conf "spark.shuffle.memoryFraction=0.8"
--conf "spark.driver.maxResultSize=16g"
--conf "spark.driver.cores=10"
--conf "spark.driver.memory=10g"

Can anyone tell me any more configs to circumvent this "executor lost" and
"executor lost failure" error?

-- 
Thank You

Regards

Punit Naik

Mime
View raw message