hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-2479) Log more Hadoop task counter values in the MapRedStats class.
Date Wed, 05 Oct 2011 18:08:30 GMT

    [ https://issues.apache.org/jira/browse/HIVE-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121304#comment-13121304
] 

jiraposter@reviews.apache.org commented on HIVE-2479:
-----------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2167/#review2355
-----------------------------------------------------------



trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java
<https://reviews.apache.org/r/2167/#comment5419>

    We may get NPE, so use code like this:
    
    Counter ctr = null;
    if(counters != null) {
      ctr= counters.findCounter(...);
      if (ctr != null) {
        ......
      }
    }
    Or put this to a common method. 
    
    


- Yongqiang


On 2011-10-05 17:58:04, Kevin Wilfong wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2167/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-10-05 17:58:04)
bq.  
bq.  
bq.  Review request for hive, Ramkumar Vadali and Yongqiang He.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  I added the counters mentioned in the task to the MapRedStats class, and modified HadoopJobExecHelper
to collect them.
bq.  
bq.  I got tired of writing the same code over and over again, so I modified the way MapRedStats
and HadoopJobExecHelper treat task counters.  MapRedStats now has an enum with all of the
task counters we want to collect, it is a subset of the enum in Task$Counter.  Task is package
private so the enum in it is unavailable.  MapRedStats now contains a map from the enum values
to the values of the counters, if they were set.  HadoopJobExecHelper loops over the enum
values and tries to get a value for each counter.  As long as the new getter and setter methods
are used the functionality is the same, in particular for the getter, if a counter was set,
it returns the value of the counter, otherwise it returns -1.
bq.  
bq.  
bq.  This addresses bug Hive-2479.
bq.      https://issues.apache.org/jira/browse/Hive-2479
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 1178612 
bq.    trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1178612 
bq.  
bq.  Diff: https://reviews.apache.org/r/2167/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  I ran some queries to verify the counters were being populated.
bq.  
bq.  I also ran a few of the unit test queries to verify I hadn't broken anything.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Kevin
bq.  
bq.


                
> Log more Hadoop task counter values in the MapRedStats class.
> -------------------------------------------------------------
>
>                 Key: HIVE-2479
>                 URL: https://issues.apache.org/jira/browse/HIVE-2479
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Kevin Wilfong
>            Assignee: Kevin Wilfong
>         Attachments: HIVE-2479.1.patch.txt, HIVE-2479.2.patch.txt, HIVE-2479.3.patch.txt
>
>
> We should log more of the Hadoop task tracker counters in the MapRedStats class, in order
to make them available to hooks and improve logging.
> We should make the counters that are logged configurable, so that if different Hadoop
versions are used, different counters can be collected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message