chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Huang, Jie" <>
Subject RE: ShuffleInput is not under the "mapred" context since Hadoop 1.0 metrics2
Date Tue, 17 Jul 2012 02:51:30 GMT
We'd better to do the modification in Hadoop side, like:

	  final MetricsRegistry registry = new MetricsRegistry("shuffleInput");
      final MetricMutableCounterLong inputBytes =
          registry.newCounter("shuffle_input_bytes", "", 0L);
      final MetricMutableCounterInt failedFetches =
          registry.newCounter("shuffle_failed_fetches", "", 0);
      final MetricMutableCounterInt successFetches =
          registry.newCounter("shuffle_success_fetches", "", 0);
      private volatile int threadsBusy = 0;

      ShuffleClientInstrumentation(JobConf conf) {
        registry.setContext("mapred")  		<<< fix point
				.tag("user", "User name", conf.getUser())
                .tag("jobName", "Job name", conf.getJobName())
                .tag("jobId", "Job ID", ReduceTask.this.getJobID().toString())
                .tag("taskId", "Task ID", getTaskID().toString())
                .tag("sessionId", "Session ID", conf.getSessionId());

Thank you && Best Regards,
Grace (Huang Jie)

-----Original Message-----
From: Ariel Rabkin [] 
Sent: Tuesday, July 17, 2012 10:33 AM
Subject: Re: ShuffleInput is not under the "mapred" context since Hadoop 1.0 metrics2

Aha.  Do you have a proposed fix for the problem?


On Mon, Jul 16, 2012 at 10:16 PM, Huang, Jie <> wrote:
> Hi all,
> Trying to upgrade Hadoop to 1.0.x, we found that the shuffleInput metric (for each reduce
task) is not under the “mapred” context, according to the following implementation in Currently, the ShuffleInput metric is under “default” context, since
it is not set explicitly.
> Consequently, we cannot have “Hadoop_mapred_shuffleInput” directory under repos/*/

Ari Rabkin
Princeton Computer Science Department
View raw message