hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Smith <psm...@aconex.com>
Subject Re: What do you use for capturing Disk I/O?
Date Mon, 14 Dec 2009 22:59:23 GMT
If you want rich detailed metrics from the OS, and hardware such as IO rates, I would HIGHLY
recommend SGI's open-source Performance Co-Pilot (PCP)

http://oss.sgi.com/projects/pcp/

It really hasn't been given the visibility many other projects have but having used this for
5 years now I can't live with out it.  SGI are obviously large scale, and PCP was designed
from the ground up to deal with high volumes, it works for NASA's Columbia:

http://www.nas.nasa.gov/About/Projects/Columbia/columbia.html

You can capture archives of data and go over it retrospectively in fine detail at any point,
looking for correlation between metrics.

cheers,

Paul
On 11/12/2009, at 12:40 PM, Matt Massie wrote:

> If you're looking for ganglia gmetric scripts for Disk I/O, take a look at
> http://ganglia.info/gmetric/ or http://ben.hartshorne.net/ganglia/.  At the
> very bottom of Ben's page you'll find disk_metric.sh.
> 
> -Matt
> 
> On Thu, Dec 10, 2009 at 4:55 PM, Andy Sautins
> <andy.sautins@returnpath.net>wrote:
> 
>> 
>>   I have a question that I got an interesting and helpful answer for on
>> the IRC channel today, but thought I'd open it up to a larger group as well.
>> 
>>  My problem is hopefully a very common problem.  I'm using Ganglia for
>> trend graphing.  We are using Ganglia 3.0.7 for convenience since RPMS are
>> available for CentOS 5.4 through EPEL.  Everything generally works great,
>> but Ganglia doesn't appear to gather Disk I/O stats ( writes/sec, reads/sec,
>> etc ) as a metric.  Some quick looking on the web seemed to confirm that
>> ganglia doesn't capture disk I/O stats by default, but talk about how it's
>> generally pretty easy to capture using gmetric.
>> 
>>  I guess my question would be is there a common way of doing this?  What
>> do you use?  Do you find Disk I/O to be a useful/necessary metric?
>> 
>>  Any thoughts would be much appreciated.
>> 
>> 


Mime
View raw message