hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: What do you use for capturing Disk I/O?
Date Mon, 14 Dec 2009 23:21:20 GMT
On Mon, Dec 14, 2009 at 5:59 PM, Paul Smith <psmith@aconex.com> wrote:
> If you want rich detailed metrics from the OS, and hardware such as IO rates, I would
HIGHLY recommend SGI's open-source Performance Co-Pilot (PCP)
>
> http://oss.sgi.com/projects/pcp/
>
> It really hasn't been given the visibility many other projects have but having used this
for 5 years now I can't live with out it.  SGI are obviously large scale, and PCP was designed
from the ground up to deal with high volumes, it works for NASA's Columbia:
>
> http://www.nas.nasa.gov/About/Projects/Columbia/columbia.html
>
> You can capture archives of data and go over it retrospectively in fine detail at any
point, looking for correlation between metrics.
>
> cheers,
>
> Paul
> On 11/12/2009, at 12:40 PM, Matt Massie wrote:
>
>> If you're looking for ganglia gmetric scripts for Disk I/O, take a look at
>> http://ganglia.info/gmetric/ or http://ben.hartshorne.net/ganglia/.  At the
>> very bottom of Ben's page you'll find disk_metric.sh.
>>
>> -Matt
>>
>> On Thu, Dec 10, 2009 at 4:55 PM, Andy Sautins
>> <andy.sautins@returnpath.net>wrote:
>>
>>>
>>>   I have a question that I got an interesting and helpful answer for on
>>> the IRC channel today, but thought I'd open it up to a larger group as well.
>>>
>>>  My problem is hopefully a very common problem.  I'm using Ganglia for
>>> trend graphing.  We are using Ganglia 3.0.7 for convenience since RPMS are
>>> available for CentOS 5.4 through EPEL.  Everything generally works great,
>>> but Ganglia doesn't appear to gather Disk I/O stats ( writes/sec, reads/sec,
>>> etc ) as a metric.  Some quick looking on the web seemed to confirm that
>>> ganglia doesn't capture disk I/O stats by default, but talk about how it's
>>> generally pretty easy to capture using gmetric.
>>>
>>>  I guess my question would be is there a common way of doing this?  What
>>> do you use?  Do you find Disk I/O to be a useful/necessary metric?
>>>
>>>  Any thoughts would be much appreciated.
>>>
>>>
>
>

A modern (linux/unix) operating system provides this information for
each disk via SNMP:

I plot both the bytes read/written and # of transactions for each
disk. Be warned if you are doing very very very heavy disk usage this
value can wrap around in a 5 minute polling interval, you might have
to go to a 1 minute interval.

 <oid_index>.1.3.6.1.4.1.2021.13.15.1.1.1</oid_index>

        <fields>
                <hrDiskIOIndex>
                        <name>Index</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.4.1.2021.13.15.1.1.1</oid>
                </hrDiskIOIndex>
                <hrDiskIODescr>
                        <name>Description</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.4.1.2021.13.15.1.1.2</oid>
                </hrDiskIODescr>
                <hrDiskIONRead>
                        <name>Number of Bytes Read</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.13.15.1.1.3</oid>
                </hrDiskIONRead>
                <hrDiskIONWrite>
                        <name>DiskIO Number of Bytes Written</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.13.15.1.1.4</oid>
                </hrDiskIONWrite>
                <hrDiskIOReads>
                        <name>Number of Read accesses</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.13.15.1.1.5</oid>
                </hrDiskIOReads>
                <hrDiskIOWrites>
                        <name>Number of Write accesses</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.13.15.1.1.6</oid>
                </hrDiskIOWrites>
        </fields>

Edward

Mime
View raw message