hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.
Date Tue, 21 Jun 2016 17:23:57 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342237#comment-15342237
] 

Sangjin Lee commented on MAPREDUCE-6720:
----------------------------------------

If we are going to store the map counters and reduce counters, we should make the metric names
such that one can easily query for them only. So in that sense, making it a total prefix or
suffix to the metric name might be a better idea? In other words, how about something like

{noformat}
org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for total.
MAP:org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for MAP.
REDUCE:org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for REDUCE.
{noformat}

or 

{noformat}
org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for total.
org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS:MAP for MAP.
org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS:REDUCE for REDUCE.
{noformat}
?

> Inconsistent values of counters across tasks and job reported to timeline service.
> ----------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6720
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: applicationmaster
>    Affects Versions: YARN-2928
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>              Labels: yarn-2928-1st-milestone
>         Attachments: Counters For Job.png, Counters for Tasks.png, MAPREDUCE-6720-YARN-2928.01.patch,
MAPREDUCE-6720-YARN-2928.02.patch
>
>
> While testing found below issue. For some of the task counters, we do not have consistent
values. This is not the case with every counter though.
> Consider the case of counter "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps associated
with this flow run, the values were 312006 bytes each (which equals to value for a flow run
i.e. 3 * 312006 = 936018). Drilling further down I found though that for one of the apps,
the 4 tasks(2 mappers and 2 reducers) had values as 155918 bytes each for the 2 reducers and
156003 bytes each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918) or 623842
bytes but it is only 312006 bytes which indicates that only counter value of mappers is being
picked up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message