hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Surówka <>
Subject Review Request: Introduction of client statistics publishers possibility
Date Wed, 14 Sep 2011 22:19:21 GMT

This is an automatically generated e-mail. To reply, visit:

Review request for hive and Ning Zhang.


The purpose of this change is to allow publication or storage of counters while the job is

Introduced two new variables to hive-default.xml and "hive.client.stats.publishers"
and "hive.client.stats.counters". First one specifies classes names, whose instances will
be executed by (similarly as hooks are) in its method progress(ExecDriverTaskHandle):
MapRedStats. Second one specifies list of counters that any client stat publishers should
publish or stored. Details regarding format of this list is up to a specific deployment (it
is up to client stats publishers to parse it), yet it is required to use display names of
counter groups and counters.

Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must
be implemented by all stats publishers.

Added code to progress(ExecDriverTaskHandle): MapRedStats from that
puts counters' values to a Java map and then executes registered client stats publishers giving
them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters)
and getClientStatsPublishers() that are used by code from previous sentence.

Made cosmetic changes in two other classes

This addresses bug HIVE-2446.


  trunk/common/src/java/org/apache/hadoop/hive/conf/ 1170867 
  trunk/conf/hive-default.xml 1170867 
  trunk/metastore/src/test/org/apache/hadoop/hive/metastore/ 1170867

  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ 1170867 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/history/ 1170867 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ PRE-CREATION



Run some random tests, and still running the unit tests.



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message