hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-6105) Support for new REST end point /clusterids
Date Mon, 23 Jan 2017 13:19:26 GMT

    [ https://issues.apache.org/jira/browse/YARN-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834435#comment-15834435
] 

Varun Saxena edited comment on YARN-6105 at 1/23/17 1:19 PM:
-------------------------------------------------------------

bq. I discussed with cloud team today, constant clusterId or storing it separately as info
does not work for them.
Hmm...Trying to understand the issue better. What issues do they face if they publish entities
with a constant cluster ID in the cloud? Is there any advantage in differentiating between
different ephemeral clusters, and that too as part of the context(i.e. via YARN tags) ? In
a cloud scenario, does the user even care about the different cluster IDs' each time his job
is run? Wouldn't cluster information be provided by the Cloud Manager or we plan to have Cloud
Manager get this info too from ATS? Agree that each cluster is unique, but if cluster ID is
dynamically generated at runtime as clusters come up and go away, we need to see how much
sense this cluster ID would make for end user. 
Anyways, assuming that for cloud scenario, a constant cluster is not to be given,  if we want
a list of all cluster IDs', I guess we would need it at real time. In that case we cannot
have a periodic job finding out unique cluster IDs' and adding them to the new table as suggested
above. Correct ?

I am fine with having a /clusterids per say as I mentioned on the other JIRA, but for cloud
use case, wouldn't this list become quite large? And as it becomes quite large, we need to
see how useful it is.

We can probably provide some API to register cluster from an off app client if the use case
for cloud is strong enough and there is no feasible alternative. Thoughts ?
We would also need to take care of ordering then as well.  


was (Author: varun_saxena):
bq. I discussed with cloud team today, constant clusterId or storing it separately as info
does not work for them.
Hmm...Trying to understand the issue better. What issues do they face if they publish entities
with a constant cluster ID in the cloud? Is there any advantage in differentiating between
different ephemeral clusters, and that too as part of the context(i.e. via YARN tags) ? In
a cloud scenario, does the user even care about the different cluster IDs' each time his job
is run? Wouldn't cluster information be provided by the Cloud Manager or we plan to have Cloud
Manager get this info too from ATS? Agree that each cluster is unique, but if cluster ID is
dynamically generated at runtime as clusters come up and go away, we need to see how much
sense this cluster ID would make for end user. 
Anyways, assuming that for cloud scenario, a constant cluster is not to be given,  if we want
a list of all cluster IDs', I guess we would need it at real time. In that case we cannot
have a periodic job finding out unique cluster IDs' and adding them to the new table as suggested
above. Correct ?

I am fine with having a /clusterids per say as I mentioned on the other JIRA, but for cloud
use case, wouldn't this list become quite large? And as it becomes quite large, we need to
see how useful it is.

We can probably provide some API to register cluster if the use case for cloud is strong enough
and there is no feasible alternative. Thoughts ?

> Support for new REST end point /clusterids
> ------------------------------------------
>
>                 Key: YARN-6105
>                 URL: https://issues.apache.org/jira/browse/YARN-6105
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Rohith Sharma K S
>
> As discussed in YARN-5378 and YARN-6095, it is required to have */clusterids* that returns
list of clusterids that back end has is useful. 
> Use case : In cloud, clusters are arbitrarily spin up and destroyed. Each cluster has
its own clusterId which UI never knows about it. To all those newly spin up cluster, same
ATS server has been used. And sam web UI has been used. Admin can select the clusterId and
navigate to any pages. So, it is worth to list ClusterId's from ATS



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message