hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@apache.org>
Subject Re: Hadoop tool-kit for monitoring
Date Tue, 17 May 2011 22:15:30 GMT

On May 17, 2011, at 3:11 PM, Mark question wrote:

> So what other memory consumption tools do you suggest? I don't want to do it
> manually and dump statistics into file because IO will affect performance
> too.

	We watch memory with Ganglia.  We also tune our systems such that a task will only take X
amount.  In other words, given an 8gb RAM:

	1gb for the OS
	1gb for the TT and DN
	6gb for all tasks

	if we assume each task will take max 1gb, then we end up with 3 maps and 3 reducers.  

	Keep in mind that the mem consumed is more than just JVM heap size.
View raw message