hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3970) Counters written to the job history cannot be recovered back
Date Fri, 22 Aug 2008 09:40:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624635#action_12624635

Amar Kamat commented on HADOOP-3970:

It seems that the jobhistory uses the display names instead of actual names in the stringified

One such example of actual name and display name is 
_actual-name_ = {{HDFS_WRITE}} and _display-name_ = {{HDFS bytes written}}. 

Note that the mapping from the actual name to the display name is mentioned in the _.properties_

Looks like its difficult to get the (actual) name from the display name because of the following
1)  Display names can be same across counters. 
2) There are two ways to add/increment a counter 
 - pass the _group_, _counter_ and the _value_
 - pass the (enum) _key_ and the _value_. For the (enum) key, the name is the classname for
that (enum) _key_.
The display name is obtained from the _property_ file. In case there is no property file corresponding
to the enum, the actual name is used. In either case, figuring out the actual name seems difficult.

So, I suggest we also add the actual name to jobhistory along with the display name and provide
api to set the display-name of the counter/group.

> Counters written to the job history cannot be recovered back
> ------------------------------------------------------------
>                 Key: HADOOP-3970
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3970
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Amar Kamat
>         Attachments: HADOOP-3970-v1.patch
> Counters that are written to the JobHistory are stringified using {{Counters.makeCompactString()}}.
The format in which this api converts the counter into a string is _groupname.countername:value_.
The problem is that _groupname_ and _countername_ can contain a '.' and hence recovering the
counter becomes difficult. Since JobHistory can be used for various purposes, reconstructing
the counter object back might be useful. One such usecase is HADOOP-3245. There should be
some way to recover the counter object back from its string representation and also to keep
the string version readable.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message