spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Tanase <>
Subject Spark Streaming scheduler delay VS driver.cores
Date Sat, 17 Oct 2015 19:58:18 GMT

I’ve recently bumped up the resources for a spark streaming job – and the performance
started to degrade over time.
it was running fine on 7 nodes with 14 executor cores each (via Yarn) until I bumped executor.cores
to 22 cores/node (out of 32 on AWS c3.xlarge, 24 for yarn)

The driver has 2 cores and 2 GB ram (usage is at zero).

For really low data volume it goes from 1-2 seconds per batch to 4-5 s/batch after about 6
hours, doing almost nothing. I’ve noticed that the scheduler delay is 3-4s, even 5-6 seconds
for some tasks. Should be in the low tens of milliseconds. What’s weirder is that under
moderate load (thousands of events per second) - the delay is not as obvious anymore.

After this I reduced the executor.cores to 20 and bumped driver.cores to 4 and it seems to
be ok now.
However, this is totally empirical, I have not found any documentation, code samples or email
discussion on how to properly set driver.cores.

Does anyone know:

  *   If I assign more cores to the driver/application manager, will it use them?
     *   I was looking at the process list with htop and only one of the jvm’s on the driver
was really taking up CPU time
  *   What is a decent parallelism factor for a streaming app with 10-20 secs batch time?
I found it odd that at  7 x 22 = 154 the driver is becoming a bottleneck
     *   I’ve seen people recommend 3-4 taks/core or ~1000 parallelism for clusters in the
tens of nodes

Thanks in advance,
View raw message