hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Wilfong" <kevinwilf...@fb.com>
Subject Review Request: Log more Hadoop task counter values in the MapRedStats object.
Date Tue, 04 Oct 2011 17:23:03 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2167/
-----------------------------------------------------------

Review request for hive, Ramkumar Vadali and Yongqiang He.


Summary
-------

I added the counters mentioned in the task to the MapRedStats class, and modified HadoopJobExecHelper
to collect them.

I got tired of writing the same code over and over again, so I modified the way MapRedStats
and HadoopJobExecHelper treat task counters.  MapRedStats now has an enum with all of the
task counters we want to collect, it is a subset of the enum in Task$Counter.  Task is package
private so the enum in it is unavailable.  MapRedStats now contains a map from the enum values
to the values of the counters, if they were set.  HadoopJobExecHelper loops over the enum
values and tries to get a value for each counter.  As long as the new getter and setter methods
are used the functionality is the same, in particular for the getter, if a counter was set,
it returns the value of the counter, otherwise it returns -1.


This addresses bug Hve-2479.
    https://issues.apache.org/jira/browse/Hve-2479


Diffs
-----

  trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 1178612 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1178612 

Diff: https://reviews.apache.org/r/2167/diff


Testing
-------

I ran some queries to verify the counters were being populated.

I also ran a few of the unit test queries to verify I hadn't broken anything.


Thanks,

Kevin


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message