hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5551) Ignore deleted file mapping from memory computation when smaps is enabled
Date Thu, 25 Aug 2016 22:41:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437858#comment-15437858
] 

Gopal V commented on YARN-5551:
-------------------------------

bq. The more I think about this, the more I feel ignoring deleted files is the wrong thing
to do

Yes, deleted files is a red-herring (happens to be how we secure the files away from other
users).

I think the original problem of YARN killing a process needs to be fixed (the original SMAPS
fix was for HDFS Zero Copy read via mmap).

{code}
              total +=
                  Math.min(info.sharedDirty, info.pss) + info.privateDirty
                      + info.privateClean;
{code}

If as [~nroberts] suggests, If YARN counted only  the "anonymous" pages as the "will be free'd
a kill" memory, it would give me a better way.

bq. the write() case is going to eventually be throttled by the OS because it will only allow
so many dirty buffer cache pages in the system. I don't believe that's the case for the mmap'd
file.

Once you exceed the dirty_ratio, the only way you can avoid a page-fault is by modifying an
existing dirty page over & over again.

If I understand page-writeback.c correctly, the blocking operation would be the page fault
on a memory block which is missing in memory.

bq. that significant memory use needs to be associated with that process in the accounting.

Accounting isn't the problem, killing processes is the problem.

> Ignore deleted file mapping from memory computation when smaps is enabled
> -------------------------------------------------------------------------
>
>                 Key: YARN-5551
>                 URL: https://issues.apache.org/jira/browse/YARN-5551
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>            Priority: Minor
>         Attachments: YARN-5551.branch-2.001.patch
>
>
> Currently deleted file mappings are also included in the memory computation when SMAP
is enabled. For e.g
> {noformat}
> 7f612004a000-7f612004c000 rw-s 00000000 00:10 4201507513                 /dev/shm/HadoopShortCircuitShm_DFSClient_NONMAPREDUCE_-521969216_162_734673185
(deleted)
> Size:                  8 kB
> Rss:                   4 kB
> Pss:                   2 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          4 kB
> Private_Clean:         0 kB
> Private_Dirty:         0 kB
> Referenced:            4 kB
> Anonymous:             0 kB
> AnonHugePages:         0 kB
> Swap:                  0 kB
> KernelPageSize:        4 kB
> MMUPageSize:           4 kB
> 7fbf28000000-7fbf68000000 rw-s 00000000 08:02 11927571                   /tmp/7298569189125604642/arena-1291157252088664681.cache
(deleted)
> Size:            1048576 kB
> Rss:               17288 kB
> Pss:               17288 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          0 kB
> Private_Clean:       232 kB
> Private_Dirty:     17056 kB
> Referenced:        17288 kB
> Anonymous:             0 kB
> AnonHugePages:         0 kB
> Swap:                  0 kB
> KernelPageSize:        4 kB
> MMUPageSize:           4 kB
> {noformat}
> It would be good to exclude these from getSmapBasedRssMemorySize() computation.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message