hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amareshwari Sriramadasu <amar...@yahoo-inc.com>
Subject Re: Reducing Hadoop Logs
Date Wed, 10 Dec 2008 03:56:50 GMT
Arv Mistry wrote:
>  
> I'm using hadoop 0.17.0. Unfortunately I cant upgrade to 0.19.0 just
> yet.
>
> I'm trying to control the amount of extraneous files. I noticed there
> are the following log files produced by hadoop;
>
> 	On Slave
> 		- userlogs (for each map/reduce job)
> 			- stderr
> 			- stdout
> 			- syslog
> 		- datanode .log file
> 		- datanode .out file
> 		- tasktracker .log file
> 		- tasktracker .out file
>
> 	On Master
> 		- jobtracker .log file
> 		- jobtracker .out file
> 		- namenode   .log file
> 		- namenode   .out file
> 		- secondarynamenode .log file
> 		- secondarynamenode .out file		
> 		- job .xml file
> 		- history
> 			- xml file for job
>
> 	
> Does any body know of how to configure hadoop so I don't have to delete
> these files manually? Or just so that they don't get created at all.
>
> For the history files, I set hadoop.job.history.user.location to none in
> the hadoop-site.xml file but I still get the history files created.
>   
Setting hadoop.job.history.user.location to "none", makes only history 
location specified for user. JT still has history location. History will 
be cleanup after a month.

Userlogs will be cleaned up after "mapred.userlog.retain.hours", by 
default , 24hrs.

Thanks
Amareshwari
> Also I set in the log4j.properties the hadoop.root.logger=WARN but I
> still see INFO messages in datanode,jobtracker etc logs
>
> Thanks, in advance
>
> Cheers Arv
>   


Mime
View raw message