hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sagar_shukla <sagar_shu...@persistent.co.in>
Subject RE: Tracking Metrics in Hadoop by User
Date Wed, 10 Mar 2010 05:41:28 GMT
Hi Steve,
      I had observed issues with Ganglia in terms of refresh of data when the nodes go down
or removed from the cluster. It could be because of the complexity of the environment, but
I found Nagios useful in that front.

There is a Hadoop plugin available for Nagios which provides node-based statistics. Though
I have not used it, but you can give it a try and see if that is useful in providing the details
that you want.
http://exchange.nagios.org/directory/Plugins/Others/check_hadoop%252Ddfs-2Esh/details

Thanks,
Sagar Shukla

-----Original Message-----
From: Stephen Watt [mailto:swatt@us.ibm.com] 
Sent: Tuesday, March 09, 2010 11:37 PM
To: common-user@hadoop.apache.org
Subject: Tracking Metrics in Hadoop by User

I'm interested in the ability to track metrics (such as CPU time, storage 
used per machine, across the cluster) in Hadoop by User. I've taken a look 
at the Fair and Capacity Schedulers and they seem oriented towards 
ensuring fair use between users' jobs rather than providing a feature 
which also reports what resources the users actually used on the cluster. 
Likewise, with other tools like Ganglia, which appear to be concerned with 
reporting metrics by machine (and not by job). I've also taken a look 
through the common/metrics tickets in JIRA and there does not seem to be 
any open work that addresses this requirement. 

Have I missed something ? Has anyone been able to do this ? Is there a way 
to capture metrics by Job (which could be correlated back to a user?) If 
not, is there any current or forecasted work in the project that addresses 
this requirement ? 

Kind regards
Steve Watt

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent
Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed.
If you are not the intended recipient, you are not authorized to read, retain, copy, print,
distribute or use this message. If you have received this communication in error, please notify
the sender and delete all copies of this message. Persistent Systems Ltd. does not accept
any liability for virus infected mails.

Mime
View raw message