hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Lu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
Date Thu, 28 Apr 2016 20:56:13 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262973#comment-15262973

Li Lu commented on MAPREDUCE-6424:

Hi [~Naganarasimha], thanks for the work! I took a look at the patch. Most part of the patch
LGTM. One general concern: given the fact that we're returning null for most of the HistoryEvents,
maybe there's a better way to do this? My proposal would be generalize the toTimelineEvent
method into something like addDataToEntity, so that we can attach any kinds of timeline v2
data to entities (info, config, metric, event, etc,.) in different kinds of events. Due to
the nature of those HistoryEvents, it's quite possible that each event may only touch a part
of a timeline entity, so maybe we don't want to generate events and/or metrics separately?
I'm raising this point also because the MapReduce code may act as an example for how to use
the v2 APIs in general, so we may want to figure out a "scalable" way to handle all cases.
I'm terribly sorry not getting back on this point earlier. Do we still have time to find a
solution here given the time is quite limited to our 1st milestone? Thanks!

l.1053 style, parameter timestamp not used? 
l.1223 I agree with Sangjin's comments that we may probably want to use a blocking call to
post final data. On the performance side, however, is it possible to post all contents in
one batch, instead of making two calls? 

> Store MR counters as timeline metrics instead of event
> ------------------------------------------------------
>                 Key: MAPREDUCE-6424
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>            Reporter: Junping Du
>            Assignee: Naganarasimha G R
>              Labels: yarn-2928-1st-milestone
>         Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, MAPREDUCE-6424-YARN-2928.v1.002.patch,
> In MAPREDUCE-6327, we make map/reduce counters get encoded from JobFinishedEvent as timeline
events with counters details in JSON format. 
> We need to store framework specific counters as metrics in timeline service to support
query, aggregation, etc.

This message was sent by Atlassian JIRA

View raw message