spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Davies Liu (JIRA)" <>
Subject [jira] [Commented] (SPARK-10342) Cooperative memory management
Date Thu, 01 Oct 2015 22:22:27 GMT


Davies Liu commented on SPARK-10342:

This will be used internal for SQL. For example, aggregation and sort-merge-join both will
acquire large page to do in-memory aggregation or sorting, one could use most of the memory,
then the other once can't have enough memory to work. Currently, each operator will preserve
a page to make sure that they can start (could have to work with the only one page). The better
solution could be, when one operator (for example aggregation) need more memory, other operators
could be notified to release some memory by spilling. This could improve the memory utilization
(don't need to preserve a page anymore) and void OOM.

> Cooperative memory management
> -----------------------------
>                 Key: SPARK-10342
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core, SQL
>    Affects Versions: 1.5.0
>            Reporter: Davies Liu
>            Priority: Critical
> We have memory starving problems for a long time, it become worser in 1.5 since we use
larger page.
> In order to increase the memory usage (reduce unnecessary spilling) also reduce the risk
of OOM, we should manage the memory in a cooperative way, it means all the memory consume
should be also responsive to release memory (spilling) upon others' requests.
> The requests of memory could be different, hard requirement (will crash if not allocated)
or soft requirement (worse performance if not allocated). Also the costs of spilling are also
different. We could introduce some kind of priority to make them work together better.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message