hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
Date Thu, 20 Mar 2014 13:38:47 GMT

    [ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941725#comment-13941725

Rajesh Balamohan commented on YARN-1775:

- Sure, it would be great if this can be accommodated in ProcfsBasedProcessTree itself.  I
will add a config flag to enable/disable smap based computation (default is disabled).  I
will upload the patch soon.
- To get realistic RSS of a process, we need Private_Clean + Private_Dirty + Shared_Dirty
of the process (i.e shared_dirty/n where n is the number of processes sharing).  If we sum
up all Shared_Dirty of memory regions, we would be double counting across processes.  If we
take PSS, then we would end up counting Shared_Clean as well.  So closer approximation of
real shared_dirty can be obtained by doing Min(shared_dirty, PSS) = Min (shared_dirty, (shared_dirty+shared_clean)/n);
 This will fall somewhere between shared_dirty & shared_dirty/n.  Hence, closer approximation
of RSS = Math.min(sharedDirty, pss) + privateDirty + privateClean.

Also interesting would be confirming that containers still get killed for exceeding the limit
with private/non-shared pages.
Had a offline discussion with Gopal.  Based on the suggestion, I tried to ByteBuffer.allocateDirect()
which would create enough PRIVATE_DIRTY in the process.  The process got killed once it exceeded
the physical memory limits of the container.

> Create SMAPBasedProcessTree to get PSS information
> --------------------------------------------------
>                 Key: YARN-1775
>                 URL: https://issues.apache.org/jira/browse/YARN-1775
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>         Attachments: YARN-1775-v2.patch, yarn-1775-2.4.0.patch
> Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use
of PSS for computing the memory usage. 

This message was sent by Atlassian JIRA

View raw message