hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Wilfong" <>
Subject Re: Review Request: Log more Hadoop task counter values in the MapRedStats object.
Date Wed, 05 Oct 2011 17:58:04 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated 2011-10-05 17:58:04.685771)

Review request for hive, Ramkumar Vadali and Yongqiang He.


Thanks Yongqiang, that's a way better idea.  I implemented it, again leaving cpuMsec because
of the special logic for it in HadoopJobExecHelper, but converting all other counters to use


I added the counters mentioned in the task to the MapRedStats class, and modified HadoopJobExecHelper
to collect them.

I got tired of writing the same code over and over again, so I modified the way MapRedStats
and HadoopJobExecHelper treat task counters.  MapRedStats now has an enum with all of the
task counters we want to collect, it is a subset of the enum in Task$Counter.  Task is package
private so the enum in it is unavailable.  MapRedStats now contains a map from the enum values
to the values of the counters, if they were set.  HadoopJobExecHelper loops over the enum
values and tries to get a value for each counter.  As long as the new getter and setter methods
are used the functionality is the same, in particular for the getter, if a counter was set,
it returns the value of the counter, otherwise it returns -1.

This addresses bug Hive-2479.

Diffs (updated)

  trunk/ql/src/java/org/apache/hadoop/hive/ql/ 1178612 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ 1178612 



I ran some queries to verify the counters were being populated.

I also ran a few of the unit test queries to verify I hadn't broken anything.



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message