hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5551) Ignore deleted file mapping from memory computation when smaps is enabled
Date Thu, 25 Aug 2016 18:17:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437379#comment-15437379
] 

Gopal V commented on YARN-5551:
-------------------------------

bq.  I guess where I'm getting hung up is on the deleted part. Unless I'm mistaken, the OS
isn't going to care whether the file is deleted or not when the process still has a mapping
to it.

Actually, that's just a safety rail to cut down IO here - when the process exits, the deleted
file pages just disappear.

bq. So in that sense I don't see why we're special-casing deleted files.

We can apply this patch to all file mappings actually - the special-casing was primarily to
cut down the impact of the patch and reduce unintended consequences.

For non-deleted files, I'd like IO isolation as well (i.e the IO impact lasts past process-death),
but that's a harder problem to solve into the 2.7.x branch (definitely to be tackled in 3.x
and specifically for a modern cgroups setup).

> Ignore deleted file mapping from memory computation when smaps is enabled
> -------------------------------------------------------------------------
>
>                 Key: YARN-5551
>                 URL: https://issues.apache.org/jira/browse/YARN-5551
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>            Priority: Minor
>         Attachments: YARN-5551.branch-2.001.patch
>
>
> Currently deleted file mappings are also included in the memory computation when SMAP
is enabled. For e.g
> {noformat}
> 7f612004a000-7f612004c000 rw-s 00000000 00:10 4201507513                 /dev/shm/HadoopShortCircuitShm_DFSClient_NONMAPREDUCE_-521969216_162_734673185
(deleted)
> Size:                  8 kB
> Rss:                   4 kB
> Pss:                   2 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          4 kB
> Private_Clean:         0 kB
> Private_Dirty:         0 kB
> Referenced:            4 kB
> Anonymous:             0 kB
> AnonHugePages:         0 kB
> Swap:                  0 kB
> KernelPageSize:        4 kB
> MMUPageSize:           4 kB
> 7f6123f99000-7f6163f99000 rw-p 00000000 08:41 211419477                  /grid/4/hadoop/yarn/local/usercache/root/appcache/application_1466700718395_1249/container_e19_1466700718395_1249_01_000003/7389389356021597290.cache
(deleted)
> Size:            1048576 kB
> Rss:              637292 kB
> Pss:              637292 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          0 kB
> Private_Clean:         0 kB
> Private_Dirty:    637292 kB
> Referenced:       637292 kB
> Anonymous:        637292 kB
> AnonHugePages:         0 kB
> Swap:                  0 kB
> KernelPageSize:        4 kB
> {noformat}
> It would be good to exclude these from getSmapBasedRssMemorySize() computation.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message