hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
Date Sat, 30 Apr 2016 08:19:13 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265241#comment-15265241
] 

Varun Saxena commented on MAPREDUCE-6424:
-----------------------------------------

Test case is failing for me every time too. So looked into it.
The test case is failing because you are checking for a flag "jobMetricsFoundForAppEntity"
being true. But you are not breaking out of the while loop which is iterating over the thist
file. So this flag can be reset if any record of YARN_APPLICATION exists in thist file and
has no metrics in it.

If you break out of the loop as soon as jobMetricsFoundForAppEntity flag becomes true, test
case would pass even if we do not address issue with aggregation updates. And I guess thats
not required to be done for this test IIUC.
I do not think we need to invoke file system based timeline reader explicitly from the test
case as of now(if you do not have bandwidth to change it). It would however be better to change
it to guard against any future changes in FS based impl. But frankly FS based impl is unlikely
to change. 

There are some NPEs' in the test case output though. We can track that in a separate JIRA.

> Store MR counters as timeline metrics instead of event
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-6424
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>            Reporter: Junping Du
>            Assignee: Naganarasimha G R
>              Labels: yarn-2928-1st-milestone
>         Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, MAPREDUCE-6424-YARN-2928.v1.002.patch,
MAPREDUCE-6424-YARN-2928.v1.003.patch, MAPREDUCE-6424-YARN-2928.v1.004.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from JobFinishedEvent as timeline
events with counters details in JSON format. 
> We need to store framework specific counters as metrics in timeline service to support
query, aggregation, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message