hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3970) Counters written to the job history cannot be recovered back
Date Tue, 19 Aug 2008 20:03:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12623776#action_12623776

Amar Kamat commented on HADOOP-3970:

Currently the way counters are converted to string (in jobhistory) is as follows
So for a counter which has the following contents
We get 
{{g1.c1:v1, g1.c2:v2, g1.c3:v3, g2.c4:v4, g2.c5:v5}}

One way to overcome the problem stated above is to use the length of the names present in
the counter
So the above  counter might look like
{{[|g1|] g1 {[|c1|] c1:v1, [|c2|] c2:v2, [|c3|] c3:v3 }, [|g2|] g2 {[|c4|] c4:v4, [|c5|] c5:v5
Here |s| means length of s.
Hence the length of the name in the counter helps to correctly identify the name and helps
in counter recovery.

> Counters written to the job history cannot be recovered back
> ------------------------------------------------------------
>                 Key: HADOOP-3970
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3970
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Amar Kamat
> Counters that are written to the JobHistory are stringified using {{Counters.makeCompactString()}}.
The format in which this api converts the counter into a string is _groupname.countername:value_.
The problem is that _groupname_ and _countername_ can contain a '.' and hence recovering the
counter becomes difficult. Since JobHistory can be used for various purposes, reconstructing
the counter object back might be useful. One such usecase is HADOOP-3245. There should be
some way to recover the counter object back from its string representation and also to keep
the string version readable.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message