hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
Date Mon, 20 Apr 2015 20:34:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503551#comment-14503551
] 

Junping Du commented on YARN-3046:
----------------------------------

Thanks [~sjlee0] for review and comments!
bq. So am I correct in understanding that with this patch it emits events to the timeline
service? Are metrics emitted as well?
That's right. Sorry for missing metrics here. Can we file a separated MR metric JIRA to track
with adjusting current JIRA title a bit? May be we can make our end-to-end flow work first
and shooting MR metric/counters need involve more work that can happen next? 

bq. Why not use HierarchicalTimelineEntity for job entity and task entities? If hierarchical
timeline entities don't work for MR jobs and tasks, I don't know where it will. Shouldn't
we try to use it for the MR job/task/etc.?
I was thinking on this before, and I agree that HierarchicalTimelineEntity is the best for
describing parent-children relationship. The quick reason that we don't use HierarchicalTimelineEntity
here is because HierarchicalTimelineEntity is an abstract class, and we don't have related
Task/Job entity as implementations so far. Given Task/Job entity is MR specific concept rather
than generic application concept, it seems a little overweight to create the subclass of HierarchicalTimelineEntity
for Task/Job entities? I think TimelineEntity provide enough  flexibility to describe this
app-specific relationship (especially for describing other relationships, like: Tez's DAG),
as we don't expect every application have to add new sub entities of TimelineEntity. Isn't
it?

bq. For that matter, do we need to think about emitting task attempts, or should we consider
as the next step after this?
Nice catch! We addressed task attempt event as task event for now but it should be a separated
type of event which has Task_Attempt_ID as entity ID (point to Task_ID). Will address this
in next patch.

bq. JobHistoryEventHandler.java:775: is that change related with this JIRA, or is it something
different?
That fix a typo of v1 timeline service. I can remove it if you think the change is not necessary.

> [Event producers] Implement MapReduce AM writing some MR metrics to ATS
> -----------------------------------------------------------------------
>
>                 Key: YARN-3046
>                 URL: https://issues.apache.org/jira/browse/YARN-3046
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Junping Du
>         Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch,
YARN-3046-v1.patch, YARN-3046-v2.patch, YARN-3046-v3.patch, YARN-3046-v4.patch
>
>
> Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and
have the MR AM write the framework-specific metrics to ATS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message