hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Sasaki (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4540) Yarn property "yarn.nodemanager.localizer.cache.cleanup.interval-ms" not working as expected
Date Tue, 12 Jan 2016 01:44:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093111#comment-15093111
] 

Kai Sasaki commented on YARN-4540:
----------------------------------

Sorry for misunderstanding.
How can you put a log files without running jobs on YARN? Do you mean put or generate log
files by yourself under "/hadoop/yarn/local"?

My assumption is that you decreased "yarn.nodemanager.localizer.cache.target-size-mb" one
by one. And caches were deleted under the target-size-mb correctly. But it takes more time
than "yarn.nodemanger.localizer.cache.cleanup.interval-ms". Is it correct?

> Yarn property "yarn.nodemanager.localizer.cache.cleanup.interval-ms" not working as expected
> --------------------------------------------------------------------------------------------
>
>                 Key: YARN-4540
>                 URL: https://issues.apache.org/jira/browse/YARN-4540
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager, yarn
>    Affects Versions: 2.7.1
>         Environment: Ambari – Version 2.1.2.1
> YARN - Version 2.7.1.2.3
> HDP - Version 2.3.2.0
>            Reporter: Mark S
>            Priority: Minor
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> I manually add specific YARN configuration to reduce my YARN cache size, however it does
not get updated for the interval specified by:
> {code}
> yarn.nodemanager.localizer.cache.cleanup.interval-ms
> {code}
> I did notice that eventually my server did reduce my YARN cache size over the weekend.
> h5. Set Values (Should update size in 1 minute):
> {code}
> yarn.nodemanager.localizer.cache.target-size-mb=4096
> yarn.nodemanager.localizer.cache.cleanup.interval-ms=60000
> {code}
> h5. Default values (Should update size in 10 minutes):
> {code}
> yarn.nodemanager.localizer.cache.target-size-mb=10240
> yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000
> {code}
> h5.  Confirming YARN cache size
> {code}
> date && du -m /hadoop/yarn | sort -nr | head -n 20
> #date && du -m . | sort -nr | head -n 20
> {code}
> h5. See also:
> * https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
> * http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message