spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Imran Rashid (JIRA)" <>
Subject [jira] [Commented] (SPARK-9103) Tracking spark's memory usage
Date Fri, 25 Sep 2015 20:41:04 GMT


Imran Rashid commented on SPARK-9103:

ah, of course, sorry I made a big mistake.  I was thinking that you only need to keep the
latest max value per executor.  But of course if that max occurred before the latest stage
started, then you need to reset your counter.  And with concurrent stages, you can't simply
reset one global counter, since you need the max within every window.

Thanks for explaining it to me again!

> Tracking spark's memory usage
> -----------------------------
>                 Key: SPARK-9103
>                 URL:
>             Project: Spark
>          Issue Type: Umbrella
>          Components: Spark Core, Web UI
>            Reporter: Zhang, Liye
>         Attachments: Tracking Spark Memory Usage - Phase 1.pdf
> Currently spark only provides little memory usage information (RDD cache on webUI) for
the executors. User have no idea on what is the memory consumption when they are running spark
applications with a lot of memory used in spark executors. Especially when they encounter
the OOM, it’s really hard to know what is the cause of the problem. So it would be helpful
to give out the detail memory consumption information for each part of spark, so that user
can clearly have a picture of where the memory is exactly used. 
> The memory usage info to expose should include but not limited to shuffle, cache, network,
serializer, etc.
> User can optionally choose to open this functionality since this is mainly for debugging
and tuning.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message