hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5551) Ignore deleted file mapping from memory computation when smaps is enabled
Date Thu, 25 Aug 2016 17:50:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437316#comment-15437316
] 

Jason Lowe commented on YARN-5551:
----------------------------------

Special casing buffer cache pages is one thing, but I guess where I'm getting hung up is on
the deleted part.  Unless I'm mistaken, the OS isn't going to care whether the file is deleted
or not when the process still has a mapping to it.  Dirty pages will still be flushed to the
store, and if the now clean page is discarded to make room for something else and the process
comes back to touch it again, we need that updated stored data to recreate the page.  So in
that sense I don't see why we're special-casing deleted files.



> Ignore deleted file mapping from memory computation when smaps is enabled
> -------------------------------------------------------------------------
>
>                 Key: YARN-5551
>                 URL: https://issues.apache.org/jira/browse/YARN-5551
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>            Priority: Minor
>         Attachments: YARN-5551.branch-2.001.patch
>
>
> Currently deleted file mappings are also included in the memory computation when SMAP
is enabled. For e.g
> {noformat}
> 7f612004a000-7f612004c000 rw-s 00000000 00:10 4201507513                 /dev/shm/HadoopShortCircuitShm_DFSClient_NONMAPREDUCE_-521969216_162_734673185
(deleted)
> Size:                  8 kB
> Rss:                   4 kB
> Pss:                   2 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          4 kB
> Private_Clean:         0 kB
> Private_Dirty:         0 kB
> Referenced:            4 kB
> Anonymous:             0 kB
> AnonHugePages:         0 kB
> Swap:                  0 kB
> KernelPageSize:        4 kB
> MMUPageSize:           4 kB
> 7f6123f99000-7f6163f99000 rw-p 00000000 08:41 211419477                  /grid/4/hadoop/yarn/local/usercache/root/appcache/application_1466700718395_1249/container_e19_1466700718395_1249_01_000003/7389389356021597290.cache
(deleted)
> Size:            1048576 kB
> Rss:              637292 kB
> Pss:              637292 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          0 kB
> Private_Clean:         0 kB
> Private_Dirty:    637292 kB
> Referenced:       637292 kB
> Anonymous:        637292 kB
> AnonHugePages:         0 kB
> Swap:                  0 kB
> KernelPageSize:        4 kB
> {noformat}
> It would be good to exclude these from getSmapBasedRssMemorySize() computation.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message