hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Uma Maheswara Rao G 72686 <mahesw...@huawei.com>
Subject Re: Remote Blocked Transfer count
Date Sat, 22 Oct 2011 11:52:43 GMT
----- Original Message -----
From: Mark question <markq2011@gmail.com>
Date: Saturday, October 22, 2011 5:57 am
Subject: Remote Blocked Transfer count
To: common-user <common-user@hadoop.apache.org>

> Hello,
> 
>  I wonder if there is a way to measure how many of the data blocks 
> havetransferred over the network? Or more generally, how many times 
> where there
> a connection/contact between different machines?
> 
There is a metrics available in Hadoop. Did you check them.
The simplest way to configure Hadoop metrics is to funnel them into a user-configurable file
on the machine running the daemon.  Metrics are organized into “contexts” (Hadoop currently
uses “jvm”, “dfs”, “mapred”, and “rpc”), and each context is independently
configured
http://www.cloudera.com/blog/2009/03/hadoop-metrics/
You can view them by JMX.

> I thought of checking the Namenode log file which usually shows 
> blk_....from src= to dst ... but I'm not sure if it's correct to 
> count those lines.
> 
I wont recommend to depend on logs. Because if some one changes the log, then it will effect
your application. 
> Any ideas are helpful.
> Mark
> 
Regards,
Uma

Mime
View raw message