hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3981) support timeline clients not associated with an application
Date Wed, 31 Aug 2016 07:01:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451391#comment-15451391
] 

Rohith Sharma K S commented on YARN-3981:
-----------------------------------------

bq. We can launch collectors as separate process for this use case?
There are 2 ways. Firstly run as default service in NM? Pros is RM can make use of underlaying
NM-RM communication protocol. RM need to just track of collector address. Secondly run as
separate process? Pro's is is that admin can decide off-client-writer-cluster. Con's is RM
need to track this daemon separately at high level

bq. For storing those entities posted from clients, can we put them in the entity table, but
just leave some unknown fields empty? Will that be a concern for the storage API's semantics?
Yes, it is concern for storage API's semantics. Current schema structure for entity table
is *userName!clusterId!flowName!flowRunId!AppId!entityType!entityId*. So, key would become
*userName!clusterId!flowName!null!null!entityType!entityId* which I am not sure doe HBase
support this as a key for read/write.

> support timeline clients not associated with an application
> -----------------------------------------------------------
>
>                 Key: YARN-3981
>                 URL: https://issues.apache.org/jira/browse/YARN-3981
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Sangjin Lee
>            Assignee: Rohith Sharma K S
>              Labels: YARN-5355
>
> In the current v.2 design, all timeline writes must belong in a flow/application context
(cluster + user + flow + flow run + application).
> But there are use cases that require writing data outside the context of an application.
One such example is a higher level client (e.g. tez client or hive/oozie/cascading client)
writing flow-level data that spans multiple applications. We need to find a way to support
them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message