hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Shvachko <...@yahoo-inc.com>
Subject Re: dfs i/o stats
Date Tue, 30 Sep 2008 01:02:50 GMT
We use TestDFSIO for measuring IO performance on our clusters.
It is called a test, but in fact its a benchmark.
It runs a map-reduce job, which either writes to or reads from files
and collects statistics.

Another thing is that Hadoop automatically collects metrics.
Like number of creates, deletes, ls's etc.
Here are some links:
http://wiki.apache.org/hadoop/GangliaMetrics
http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/dfs/NameNodeMetrics.html
http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/dfs/FSNamesystemMetrics.html

Hope this is helpful.
--Konstantin

Shirley Cohen wrote:
> Hi,
> 
> I would like to measure the disk i/o performance of our hadoop cluster. 
> However, running iostat on 16 nodes is rather cumbersome. Does dfs keep 
> track of any stats like the number of blocks or bytes read and written? 
>  From scanning the api, I found a class called 
> "org.apache.hadoop.fs.FileSystem.Statistics" that could be relevant. 
> Does anyone know if this is what I'm looking for?
> 
> Thanks,
> 
> Shirley
> 

Mime
View raw message