hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Bockelman <bbock...@cse.unl.edu>
Subject Re: Monitoring hadoop?
Date Fri, 05 Jun 2009 16:37:12 GMT
Hey Anthony,

I'd add you can look at our Ganglia stats:

http://rcf.unl.edu/ganglia/?c=red-workers

http://rcf.unl.edu/ganglia/?c=red-workers&h=node123&m=load_one&r=hour&s=descending&hc=4

The second one is the node-specific view, and you can see all the  
Hadoop metrics integrated nicely with the system metrics.

Brian

On Jun 5, 2009, at 11:33 AM, Matt Massie wrote:

> Anthony-
>
> The ganglia web site is at http://ganglia.info/ with documentation  
> in a wiki at http://ganglia.wiki.sourceforge.net/.  There is also a  
> good wiki page at IBM as well http://www.ibm.com/developerworks/wikis/display/WikiPtype/ganglia

>  .  Ganglia packages are available for most distributions to help  
> with installation so make sure to grep for ganglia with your  
> favorite package manager (e.g. aptitude, yum, etc).  Ganglia will  
> give you more information about your cluster than just Hadoop  
> metrics.  You'll get CPU, load, memory, disk and network  monitoring  
> as well for free.
>
> You can see live demos of ganglia at http://ganglia.info/?page_id=69.
>
> Good luck.
>
> -Matt
>
> On Jun 5, 2009, at 7:10 AM, Brian Bockelman wrote:
>
>> Hey Anthony,
>>
>> Look into hooking your Hadoop system into Ganglia; this produces  
>> about 20 real-time statistics per node.
>>
>> Hadoop also does JMX, which hooks into more "enterprise"-y  
>> monitoring systems.
>>
>> Brian
>>
>> On Jun 5, 2009, at 8:55 AM, Anthony McCulley wrote:
>>
>>> Hey all,
>>> I'm currently tasked to come up with a web/flex-based
>>> visualization/monitoring system for a cloud system using hadoop as  
>>> part of a
>>> university research project.  I was wondering if I could elicit some
>>> feedback from all of you with regards to:
>>>
>>>
>>> - If you were an engineer of a cloud system running hadoop, what
>>> information would you be interested in capturing, viewing,  
>>> monitoring, etc?
>>> - Is there any sort of real-time stats or monitoring currently  
>>> available
>>> for hadoop?  if so, is in a web-friendly format?
>>>
>>> Thanks in advance,
>>>
>>> - Anthony
>>


Mime
View raw message