giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexandre Fonseca (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (GIRAPH-812) Make heap use configurable fraction of total container memory
Date Sun, 08 Dec 2013 04:49:35 GMT

     [ https://issues.apache.org/jira/browse/GIRAPH-812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alexandre Fonseca updated GIRAPH-812:
-------------------------------------

    Labels: yarn  (was: )

> Make heap use configurable fraction of total container memory
> -------------------------------------------------------------
>
>                 Key: GIRAPH-812
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-812
>             Project: Giraph
>          Issue Type: Improvement
>          Components: conf and scripts
>    Affects Versions: 1.1.0
>            Reporter: Alexandre Fonseca
>            Priority: Minor
>              Labels: yarn
>         Attachments: GIRAPH-812.patch
>
>
> Currently, both the Yarn ApplicationMaster and task containers are launched with a JVM
heap size matching the total memory available to the container. However, while the heap is
indeed one of the major users of memory in the JVM, other components also occupy memory which
is not included in the heap calculation: http://stackoverflow.com/questions/9725633/why-is-my-jvms-total-memory-usage-more-than-30-times-greater-than-its-xmx-value.

> With current code, if we attempt to launch containers with limited memory
> (e.g 512MB) and leave the default yarn.nodemanager.vmem-pmem-ratio of 2.1, it becomes
very easy to overflow the maximum amount of virtual memory and get failed application executions
even with simple applications such as the SimpleShortestPaths example. While we could force
users to set higher vmem-pmem ratios, I think a better option would be to code a configurable
margin/fraction of heap usage. In particular, I've always had the habit of setting heap usage
at 75% of the total memory available to the container. Doing this, I have had no problems
with excessive virtual memory.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message