hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Uma Maheswara Rao G 72686 <mahesw...@huawei.com>
Subject Re: Remote Blocked Transfer count
Date Sat, 22 Oct 2011 11:52:43 GMT
----- Original Message -----
From: Mark question <markq2011@gmail.com>
Date: Saturday, October 22, 2011 5:57 am
Subject: Remote Blocked Transfer count
To: common-user <common-user@hadoop.apache.org>

> Hello,
>  I wonder if there is a way to measure how many of the data blocks 
> havetransferred over the network? Or more generally, how many times 
> where there
> a connection/contact between different machines?
There is a metrics available in Hadoop. Did you check them.
The simplest way to configure Hadoop metrics is to funnel them into a user-configurable file
on the machine running the daemon.  Metrics are organized into “contexts” (Hadoop currently
uses “jvm”, “dfs”, “mapred”, and “rpc”), and each context is independently
You can view them by JMX.

> I thought of checking the Namenode log file which usually shows 
> blk_....from src= to dst ... but I'm not sure if it's correct to 
> count those lines.
I wont recommend to depend on logs. Because if some one changes the log, then it will effect
your application. 
> Any ideas are helpful.
> Mark

View raw message