spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Closed] (SPARK-21352) Memory Usage in Spark Streaming
Date Sun, 09 Jul 2017 13:03:01 GMT


Sean Owen closed SPARK-21352.

> Memory Usage in Spark Streaming
> -------------------------------
>                 Key: SPARK-21352
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: DStreams, Spark Submit, YARN
>    Affects Versions: 2.1.1
>            Reporter: Shubham Gupta
>              Labels: newbie
> I am trying to figure out the memory used by executors for a Spark Streaming job. For
data I am using the rest endpoint for Spark AllExecutors and just summing up the metrics totalDuration
* spark.executor.memory for every executor and then emitting the final sum as the memory usage.
> But this is coming out to be very small for application which ran whole day , is something
wrong with the logic.Also I am using dynamic allocation and executorIdleTimeout is 5 seconds.
> Also I am also assuming that if some executor was removed for due to idle timeout and
then was allocated to some other task then its totalDuration will be increased by the amount
of time took by the executor to execute this new task.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message