hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Surówka <rober...@buffalo.edu>
Subject Review Request: Introduction of client statistics publishers possibility
Date Wed, 14 Sep 2011 22:19:21 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1909/
-----------------------------------------------------------

Review request for hive and Ning Zhang.


Summary
-------

The purpose of this change is to allow publication or storage of counters while the job is
running.

Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers"
and "hive.client.stats.counters". First one specifies classes names, whose instances will
be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle):
MapRedStats. Second one specifies list of counters that any client stat publishers should
publish or stored. Details regarding format of this list is up to a specific deployment (it
is up to client stats publishers to parse it), yet it is required to use display names of
counter groups and counters.

Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must
be implemented by all stats publishers.

Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that
puts counters' values to a Java map and then executes registered client stats publishers giving
them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters)
and getClientStatsPublishers() that are used by code from previous sentence.

Made cosmetic changes in two other classes


This addresses bug HIVE-2446.
    https://issues.apache.org/jira/browse/HIVE-2446


Diffs
-----

  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1170867 
  trunk/conf/hive-default.xml 1170867 
  trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1170867

  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1170867 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 1170867 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublisher.java PRE-CREATION


Diff: https://reviews.apache.org/r/1909/diff


Testing
-------

Run some random tests, and still running the unit tests.


Thanks,

Robert


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message