hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Venner <ja...@attributor.com>
Subject hadoop 0.16.3 Gangia Metric Counter reporting does not appear to work
Date Tue, 20 May 2008 01:58:49 GMT
I first verified that when I was using the file context, I saw the 
counters in the report.
Then I switched context's to ganglia.

I also instrumented the low level code. I am hoping someone understands 
this off the top of their head as I don't want to dig into the metrics 
code to understand the difference ;)

Here is some syslog data. Note the missing name for the metric in the 
JobInProgress case.

The JobTracker uses:
        metricsRecord.incrMetric("maps_launched", numMapTasksLaunched);
        metricsRecord.incrMetric("maps_completed", numMapTasksCompleted);
        metricsRecord.incrMetric("reduces_launched", 
numReduceTasksLaunched);
        metricsRecord.incrMetric("reduces_completed", 
numReduceTasksCompleted);
        metricsRecord.incrMetric("jobs_submitted", numJobsSubmitted);
        metricsRecord.incrMetric("jobs_completed", numJobsCompleted);

While JobInProgress uses:
      for (String counter : group.getCounterNames()) {
        long value = group.getCounter(counter);
        jobMetrics.setTag("counter", group.getDisplayName(counter));
        jobMetrics.setMetric("value", (float) value);
        jobMetrics.update();


These are the job counters (JobInProgress)

2008-05-19 17:49:39,439 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric
Name: value, Type: float, Value: 36.0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,439 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric
Name: value, Type: float, Value: 20.0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,440 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric
Name: value, Type: float, Value: 0.0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,447 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric
Name: value, Type: float, Value: 432.0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,448 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric
Name: value, Type: float, Value: 0.0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,448 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric
Name: value, Type: float, Value: 18.0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,448 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric
Name: value, Type: float, Value: 576.0 to localhost/127.0.0.1:8649

These are the JobTracker counters
2008-05-19 17:49:39,449 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric
Name: jobs_completed, Type: int32, Value: 0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,449 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric
Name: jobs_submitted, Type: int32, Value: 1 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,449 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric
Name: maps_completed, Type: int32, Value: 18 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,449 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric
Name: maps_launched, Type: int32, Value: 20 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,450 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric
Name: reduces_completed, Type: int32, Value: 0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,450 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric
Name: reduces_launched, Type: int32, Value: 0 to localhost/127.0.0.1:8649

-- 
Jason Venner
Attributor - Program the Web <http://www.attributor.com/>
Attributor is hiring Hadoop Wranglers and coding wizards, contact if 
interested

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message