hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anfernee Xu <anfernee...@gmail.com>
Subject Re: what's counterpart for org.apache.hadoop.mapred.TaskTrackerMetricsInst in YARN
Date Tue, 04 Feb 2014 17:25:25 GMT
Thanks Harsh,

My target is that I want to monitor disk space usage on each node in the
cluster, if space usage reaches to a certain threshold on one node, my
scheduler will not select slot on that node, in order to do that, I have a
instrumentation class on each TT to report disk space usage to JobTracker
periodically. Now I want to move to YARN, I also notice YARN metric system:
 do you think I can use it?

For example,

1) I implement a metric source to collect disk space usage
2) write a filter to only collect this metric
3) write a sink to publish the metric to RM

Please comment.

Thanks a lot

On Tue, Feb 4, 2014 at 5:02 AM, Harsh J <harsh@cloudera.com> wrote:

> Unfortunately there isn't a direct equivalent available for plugging
> in instrumentation classes in YARN's NodeManagers currently. The
> closest I can locate is a comment hint in the NodeManagerMetrics class
> that toys with the same idea:
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java#L53
> If you'll absolutely need this in YARN, can you please file a YARN
> JIRA with the motivation on why?
> On Tue, Feb 4, 2014 at 9:37 AM, Anfernee Xu <anfernee.xu@gmail.com> wrote:
> > Hi,
> >
> > I'm migrating my application from MR v.1 to YARN, I do not know how to
> > convert my class which extends
> > org.apache.hadoop.mapred.TaskTrackerMetricsInst, I want this class to
> run as
> > TaskTracker starts up, if I migrate to YARN, I want it to run as
> NodeManager
> > starts up, so how can I make this happen?
> >
> > --
> > --Anfernee
> --
> Harsh J


View raw message