hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
Date Fri, 29 Apr 2016 21:14:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264756#comment-15264756

Naganarasimha G R commented on MAPREDUCE-6424:

I have uploaded a patch without consideration of above points as i felt though {{HistoryEvent.getTimelineEntitiesToPublish}}
would be ideal but has far too many changes  and need to consider how to handle configs too.
So thought better to postpone this decision to [~varun_saxena]'s patch. Thoughts?

bq. So in general, the penalty for making fine-grained putEntitiesAsync() calls shouldn't
be too big.
What i was considering was if we go for {{HistoryEvent.getTimelineEntitiesToPublish}}/ {{HistoryEvent.addDataToEntity}}
 approach and if config size is too big then we will not be able to address it by splitting
into multiple entities because :
* All entities passed to {{timelineClient.putEntities/putEntitiesAsync}} is published as a
single REST call, currently not considering number of entities/size.
* *HistoryEvent.addDataToEntity* approach doesn't model multiple entities which might be required
for publishing ApplicationEntity's metrics and configs. 
* Based on security concerns we might take different approach in publishing metrics and configs
as part of ApplicationEntity in YARN-3401

TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling is passing locally 
 T E S T S
Running org.apache.hadoop.mapred.TestMRTimelineEventHandling
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.195 sec - in org.apache.hadoop.mapred.TestMRTimelineEventHandling

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
would be helpful if someone validates it for me

> Store MR counters as timeline metrics instead of event
> ------------------------------------------------------
>                 Key: MAPREDUCE-6424
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>            Reporter: Junping Du
>            Assignee: Naganarasimha G R
>              Labels: yarn-2928-1st-milestone
>         Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, MAPREDUCE-6424-YARN-2928.v1.002.patch,
MAPREDUCE-6424-YARN-2928.v1.003.patch, MAPREDUCE-6424-YARN-2928.v1.004.patch
> In MAPREDUCE-6327, we make map/reduce counters get encoded from JobFinishedEvent as timeline
events with counters details in JSON format. 
> We need to store framework specific counters as metrics in timeline service to support
query, aggregation, etc.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message