hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Venner <ja...@attributor.com>
Subject Re: hadoop 0.16.3 Gangia Metric Counter reporting does not appear to work
Date Tue, 20 May 2008 17:38:49 GMT
I just submitted a Jira for this, with a patch against trunk.
Hadoop-3422 <https://issues.apache.org/jira/browse/HADOOP-3422>

If you pick up the patch, all of the metrics now have a leader in the 
name indicating which context/job is doing the reporting, and all of the 
job counters have their own rrd file and graph.


Jason Venner wrote:
> I first verified that when I was using the file context, I saw the 
> counters in the report.
> Then I switched context's to ganglia.
>
> I also instrumented the low level code. I am hoping someone 
> understands this off the top of their head as I don't want to dig into 
> the metrics code to understand the difference ;)
>
> Here is some syslog data. Note the missing name for the metric in the 
> JobInProgress case.
>
> The JobTracker uses:
>        metricsRecord.incrMetric("maps_launched", numMapTasksLaunched);
>        metricsRecord.incrMetric("maps_completed", numMapTasksCompleted);
>        metricsRecord.incrMetric("reduces_launched", 
> numReduceTasksLaunched);
>        metricsRecord.incrMetric("reduces_completed", 
> numReduceTasksCompleted);
>        metricsRecord.incrMetric("jobs_submitted", numJobsSubmitted);
>        metricsRecord.incrMetric("jobs_completed", numJobsCompleted);
>
> While JobInProgress uses:
>      for (String counter : group.getCounterNames()) {
>        long value = group.getCounter(counter);
>        jobMetrics.setTag("counter", group.getDisplayName(counter));
>        jobMetrics.setMetric("value", (float) value);
>        jobMetrics.update();
>
>
> These are the job counters (JobInProgress)
>
> 2008-05-19 17:49:39,439 INFO 
> org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: 
> value, Type: float, Value: 36.0 to localhost/127.0.0.1:8649
>
> 2008-05-19 17:49:39,439 INFO 
> org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: 
> value, Type: float, Value: 20.0 to localhost/127.0.0.1:8649
>
> 2008-05-19 17:49:39,440 INFO 
> org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: 
> value, Type: float, Value: 0.0 to localhost/127.0.0.1:8649
>
> 2008-05-19 17:49:39,447 INFO 
> org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: 
> value, Type: float, Value: 432.0 to localhost/127.0.0.1:8649
>
> 2008-05-19 17:49:39,448 INFO 
> org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: 
> value, Type: float, Value: 0.0 to localhost/127.0.0.1:8649
>
> 2008-05-19 17:49:39,448 INFO 
> org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: 
> value, Type: float, Value: 18.0 to localhost/127.0.0.1:8649
>
> 2008-05-19 17:49:39,448 INFO 
> org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: 
> value, Type: float, Value: 576.0 to localhost/127.0.0.1:8649
>
> These are the JobTracker counters
> 2008-05-19 17:49:39,449 INFO 
> org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: 
> jobs_completed, Type: int32, Value: 0 to localhost/127.0.0.1:8649
>
> 2008-05-19 17:49:39,449 INFO 
> org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: 
> jobs_submitted, Type: int32, Value: 1 to localhost/127.0.0.1:8649
>
> 2008-05-19 17:49:39,449 INFO 
> org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: 
> maps_completed, Type: int32, Value: 18 to localhost/127.0.0.1:8649
>
> 2008-05-19 17:49:39,449 INFO 
> org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: 
> maps_launched, Type: int32, Value: 20 to localhost/127.0.0.1:8649
>
> 2008-05-19 17:49:39,450 INFO 
> org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: 
> reduces_completed, Type: int32, Value: 0 to localhost/127.0.0.1:8649
>
> 2008-05-19 17:49:39,450 INFO 
> org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: 
> reduces_launched, Type: int32, Value: 0 to localhost/127.0.0.1:8649
>
-- 
Jason Venner
Attributor - Program the Web <http://www.attributor.com/>
Attributor is hiring Hadoop Wranglers and coding wizards, contact if 
interested

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message