hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.
Date Tue, 21 Jun 2016 16:40:58 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342112#comment-15342112
] 

Varun Saxena edited comment on MAPREDUCE-6720 at 6/21/16 4:40 PM:
------------------------------------------------------------------

I am fine with reporting Map and reduce counter values separately as well. But this will increase
the number of metrics being stored to upto 2 times. 
If we see a use case and have consensus on it, we can probably append _MAP or _REDUCE to the
counter group name as that can help us filter out certain metrics based on prefix (we can
specify prefixes in metricstoretrieve).

Please note that these metrics will be reported as part of task counters as well.
Thinking more over it, we probably have no way of filtering out Map and reduce tasks right
now. Maybe we can specify in info that a particular task is MAP or REDUCE.
Thoughts ?


was (Author: varun_saxena):
I am fine with reporting Map and reduce counter values separately as well. But this will increase
the number of metrics being stored to upto 2 times. 
If we see a use case and have consensus on it, we can probably append _MAP or _REDUCE to the
counter group name as that can help us filter out certain metrics based on prefix (we can
specify prefixes in metricstoretrieve)

> Inconsistent values of counters across tasks and job reported to timeline service.
> ----------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6720
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: applicationmaster
>    Affects Versions: YARN-2928
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>              Labels: yarn-2928-1st-milestone
>         Attachments: Counters For Job.png, Counters for Tasks.png, MAPREDUCE-6720-YARN-2928.01.patch
>
>
> While testing found below issue. For some of the task counters, we do not have consistent
values. This is not the case with every counter though.
> Consider the case of counter "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps associated
with this flow run, the values were 312006 bytes each (which equals to value for a flow run
i.e. 3 * 312006 = 936018). Drilling further down I found though that for one of the apps,
the 4 tasks(2 mappers and 2 reducers) had values as 155918 bytes each for the 2 reducers and
156003 bytes each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918) or 623842
bytes but it is only 312006 bytes which indicates that only counter value of mappers is being
picked up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message