hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3970) Counters written to the job history cannot be recovered back
Date Mon, 03 Nov 2008 03:02:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644637#action_12644637
] 

Matei Zaharia commented on HADOOP-3970:
---------------------------------------

Will this be a standardized format for counters in future Hadoop releases, or are there other
issues you guys think will need to be fixed? I've been working on some scripts for parsing
job history logs to compute statistics about how a Hadoop cluster is being used (see https://issues.apache.org/jira/browse/HADOOP-3708)
and I've seen small things being different in different versions of Hadoop. I think it would
be beneficial to choose a format and make it standard for Hadoop 1.0, because I imagine others
are also interested in being able to parse out info from the job history logs.

> Counters written to the job history cannot be recovered back
> ------------------------------------------------------------
>
>                 Key: HADOOP-3970
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3970
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>             Fix For: 0.19.0
>
>         Attachments: HADOOP-3970-v1.patch, HADOOP-3970-v2.patch, HADOOP-3970-v3.patch,
HADOOP-3970-v4.1.patch, HADOOP-3970-v4.patch
>
>
> Counters that are written to the JobHistory are stringified using {{Counters.makeCompactString()}}.
The format in which this api converts the counter into a string is _groupname.countername:value_.
The problem is that _groupname_ and _countername_ can contain a '.' and hence recovering the
counter becomes difficult. Since JobHistory can be used for various purposes, reconstructing
the counter object back might be useful. One such usecase is HADOOP-3245. There should be
some way to recover the counter object back from its string representation and also to keep
the string version readable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message