hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
Date Mon, 13 Apr 2015 18:46:15 GMT

    [ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14492823#comment-14492823
] 

Zhijie Shen commented on YARN-3046:
-----------------------------------

Junping, thanks for the patch. Overall it looks good to me. Some comments about the patch
details.

1. I'm not sure if we should have a MR config to determine is new or old timeline service.
If this MR config is set to true, but YARN cluster is still setup with old timeline service.
It still doesn't work.
{code}
433	  public static final String MAPREDUCE_JOB_NEW_TIMELINE_SERVICE_ENABLED =
434	      "mapreduce.job.new-timeline-service.enabled";
{code}

2. Node need to have JobHistoryEventUtils, you can move util method to JobHistoryUtils if
you want.

3. {{TimelineClient getTimelineClient();}} shouldn't belong to AppContext, which will be presented
to the history server too. It's better to put it into RunningAppContext.

4. In the current way of shutting down the threadpool, is it guaranteed that the pending entity
is going to be published before shutting down?

5. It's not documented well, but can we treat parent -> child relationship as the outbound
edge from parent and inbound edge to child. Therefore,{{ taskEntity.addIsRelatedToEntity(jobEntity)}}?
{code}
entity.addRelatesToEntity(relatedJobEntity, jobId.toString());
{code}

6. In {{TaskAttemptStartedEvent}}, TASK_TYPE is written twice.

7. createTimelineEntityFromEvent -> createJobEntity and createTimelineEntityFromEventWithRelatedJobEntity
-> createTaskEntity?

8. Use {{IOException | YarnException e}}?
{code}
1122	    try {
1123	      timelineClient.putEntities(tEntity);
1124	    } catch (IOException ex) {
1125	      LOG.error("Error putting entity " + tEntity.getId() + " to new Timeline"
1126	      + "Service", ex);
1127	    } catch (YarnException ex) {
{code}


> [Event producers] Implement MapReduce AM writing some MR metrics to ATS
> -----------------------------------------------------------------------
>
>                 Key: YARN-3046
>                 URL: https://issues.apache.org/jira/browse/YARN-3046
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Junping Du
>         Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch,
YARN-3046-v1.patch
>
>
> Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and
have the MR AM write the framework-specific metrics to ATS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message