hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-2446) Introduction of client statistics publishers possibility
Date Fri, 16 Sep 2011 17:45:10 GMT

    [ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106625#comment-13106625
] 

jiraposter@reviews.apache.org commented on HIVE-2446:
-----------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1909/#review1935
-----------------------------------------------------------

Ship it!


- Ning


On 2011-09-16 17:35:58, Robert Surówka wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1909/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-09-16 17:35:58)
bq.  
bq.  
bq.  Review request for hive and Ning Zhang.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  The purpose of this change is to allow publication or storage of counters while the job
is running.
bq.  
bq.  Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers"
and "hive.client.stats.counters". First one specifies classes names, whose instances will
be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle):
MapRedStats. Second one specifies list of counters that any client stat publishers should
publish or stored. Details regarding format of this list is up to a specific deployment (it
is up to client stats publishers to parse it), yet it is required to use display names of
counter groups and counters.
bq.  
bq.  Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that
must be implemented by all stats publishers.
bq.  
bq.  Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java
that puts counters' values to a Java map and then executes registered client stats publishers
giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters)
and getClientStatsPublishers() that are used by code from previous sentence.
bq.  
bq.  Made cosmetic changes in one other class
bq.  
bq.  
bq.  This addresses bug HIVE-2446.
bq.      https://issues.apache.org/jira/browse/HIVE-2446
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171283 
bq.    trunk/conf/hive-default.xml 1171283 
bq.    trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1171283 
bq.    trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 1171283 
bq.    trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublisher.java PRE-CREATION

bq.  
bq.  Diff: https://reviews.apache.org/r/1909/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Run some random tests, and still running the unit tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Robert
bq.  
bq.



> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
>                 Key: HIVE-2446
>                 URL: https://issues.apache.org/jira/browse/HIVE-2446
>             Project: Hive
>          Issue Type: Improvement
>          Components: Clients, Statistics
>            Reporter: Robert Surówka
>            Assignee: Robert Surówka
>            Priority: Minor
>         Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job
is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers"
and "hive.client.stats.counters". First one specifies classes names, whose instances will
be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle):
MapRedStats. Second one specifies list of counters that any client stat publishers should
publish or stored. Details regarding format of this list is up to a specific deployment (it
is up to client stats publishers to parse it), yet it is required to use display names of
counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that
must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java
that puts counters' values to a Java map and then executes registered client stats publishers
giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters)
and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message