hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arv Mistry" <...@kindsight.net>
Subject Reducing Hadoop Logs
Date Tue, 09 Dec 2008 18:51:18 GMT
 
I'm using hadoop 0.17.0. Unfortunately I cant upgrade to 0.19.0 just
yet.

I'm trying to control the amount of extraneous files. I noticed there
are the following log files produced by hadoop;

	On Slave
		- userlogs (for each map/reduce job)
			- stderr
			- stdout
			- syslog
		- datanode .log file
		- datanode .out file
		- tasktracker .log file
		- tasktracker .out file

	On Master
		- jobtracker .log file
		- jobtracker .out file
		- namenode   .log file
		- namenode   .out file
		- secondarynamenode .log file
		- secondarynamenode .out file		
		- job .xml file
		- history
			- xml file for job

	
Does any body know of how to configure hadoop so I don't have to delete
these files manually? Or just so that they don't get created at all.

For the history files, I set hadoop.job.history.user.location to none in
the hadoop-site.xml file but I still get the history files created.
Also I set in the log4j.properties the hadoop.root.logger=WARN but I
still see INFO messages in datanode,jobtracker etc logs

Thanks, in advance

Cheers Arv

Mime
View raw message