hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3034) [Aggregator wireup] Implement RM starting its ATS writer
Date Mon, 16 Feb 2015 19:19:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323154#comment-14323154

Naganarasimha G R commented on YARN-3034:

Hi [~sjlee0] & [~zjshen], thanks for reviewing the patch, 
bq. If aggregator is able to handle the requests in the async way, I'm okay to use rmcontext
dispatcher. Otherwise, let's make sure at least we're using a separate async dispatcher.
+1 for having separate async dispatcher as anyway we are not planning to handle container
events in RM

bq.  this creates a dependency from RM to the timeline service; perhaps it is unavoidable...
Based on the discussions we had on the last week I understand that RM and NM should not be
directly dependent on TimelineService .
But based on 3030 patch, BaseAggregatorService.java is in timeline service project hence where
to place this RMTimelineAggregator.java class (as it extends BaseAggregatorService ) ? 
If we plan to handle similar to current approach i.e send the Entity data through a rest client
to a timeline writer service(RMTimelineAggregator), where should this service be running i.e.
as part of which process or should it be a daemon on its own?

Other queries :
# Is RMTimelineAggregator is expected to do any primary (preliminary) aggregation of some
metrics ? Just wanted to the know reason to have a specific TimeLineAggregator for RM separately?
Similarly for NM/Applications too, what if there are no primary aggregations and just want
to push the entity data to ATS, in these cases do we require separate services handling for
per app ? 
# User and Queue Entities have been newly added in the 3041 Datamodel proposal: 
IIUC RM needs to add User and Queue Entities when application is created if the specified
user and queue doesnt exist as entity in ATS ? Apart from this  Queue Entity has Parent Queue
information, is it something like when CS/FS is initialized we need to create Entities for
new queues and hierarcies ? Is it not sufficent to just have for Leaf Queue Entity and just
have parent path as its meta info, is hierarchy req?

Based on clarification on these points, i can rework on the patch along with fixing for other
small issues.

> [Aggregator wireup] Implement RM starting its ATS writer
> --------------------------------------------------------
>                 Key: YARN-3034
>                 URL: https://issues.apache.org/jira/browse/YARN-3034
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Naganarasimha G R
>         Attachments: YARN-3034.20150205-1.patch
> Per design in YARN-2928, implement resource managers starting their own ATS writers.

This message was sent by Atlassian JIRA

View raw message