hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohith Sharma K S <rohithsharm...@huawei.com>
Subject RE: how to delete logs automatically from hadoop yarn
Date Mon, 20 Apr 2015 04:39:14 GMT
Hi

With below configuration , log deletion should be triggered.  You can see from the log that
deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below
log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler:
Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800
seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”,
default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it
is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg
Mime
View raw message