hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Hadoop Node Monitoring
Date Thu, 12 Nov 2009 14:36:19 GMT
Definatly check out my presentation above on cloudera's site link is above.

Hadoop specific counters are available. Each component namenode,
datanode, etc has counter objects associated with it.

Hadoop allows you to push statistics at ganglia so this is one nice
option. More of less once you get the configuration correct each node
will send its performance data to ganglia.

Cacti works in the other direction by polling nodes and pulling
counter information from them.

The perk of the cacti setup is that I spent some time grouping related
variables into a single graph.

The perk of the ganglia configuration is there is no per node
configuration, since it just pushes counter data.

On the cacti side I am looking to add some graphs that pull data from
the job client, currently running maps and reduces etc.

So to answer you question people use all three and they all have
access to the same data.

On 11/12/09, John Martyniak <john@beforedawnsolutions.com> wrote:
> I do already use Nagios, and have been monitoring the availability
> etc, of the network.
> But I was hoping to get more insight into the load/workings of the
> hadoop network and Ganglia seemed like a good start.
> Do you use either Ganglia or Cacti, or something else?
> -John
> On Nov 12, 2009, at 12:51 AM, Kevin Sweeney wrote:
>> Nagios is always a good start. This webcast has some good
>> information on
>> this subject:
>> http://www.cloudera
>> .com/blog/2009/11/09/hadoop-world-monitoring-best-practices-from-ed-
>> capriolo/
>> <http://www.cloudera.com/blog/2009/11/09/hadoop-world-monitoring-best-practices-from-ed-capriolo/
>> >
>> On Wed, Nov 11, 2009 at 10:46 PM, John Martyniak <
>> john@beforedawnsolutions.com> wrote:
>>> Is there a good solution for Hadoop node monitoring?  I know that
>>> Cacti and
>>> Ganglia are probably the two big ones, but are they the best ones
>>> to use?
>>> Easiest to setup? Most thorough reporting, etc.
>>> I started to play with Ganglia, and the install is crazy, I am
>>> installing
>>> it on CentOS and having all sorts of troubles.  So any idea there
>>> would be
>>> very helpful.
>>> Thank you,
>>> -John

View raw message