hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-7215) REST API to list all deployed services by the same user
Date Tue, 19 Sep 2017 22:20:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16172434#comment-16172434
] 

Eric Yang edited comment on YARN-7215 at 9/19/17 10:19 PM:
-----------------------------------------------------------

[~jianhe] How does RM handle a service that is in stopped state?  Stopped slider application
does not have any record in resource manager.  Same slider application can have multiple Application
ID when the application has been restarted.  Slider uses HDFS file to persist the paused application,
but having resource manager to crawl through lists of HDFS directories to find stopped service
seems like potential load attack to namenode.  It would be better to have the operational
record index, and cached by well known mechanism like a SOLR collection.  This also reduces
having to brew another random read/write, low latency, index, cache mechanism in YARN.  Both
HBase and SOLR have solved random read/write on top of HDFS with some success.  It would be
better to use existing libraries that have been baked for several years than inventing something
new for specialized purpose.


was (Author: eyang):
[~jianhe] How does RM handle a service that is in stopped state?  Stopped slider application
does not have any record in resource manager.  Same slider application can have multiple Application
ID when the application has been restarted.  Slider uses HDFS file to persist the paused application,
but having resource manager to crawl through lists of HDFS directories to find stopped service
seems like potential load attack to namenode.  It would be better to have the operational
record index, and cached by well known mechanism like a SOLR collection.  This also reduces
having to brew another random read/write, low latency, index, cache mechanism in YARN.  Both
HBase and SOLR have solved random read/write on top of HDFS with some success.  It would be
better to we use existing libraries that have been baked for several years than inventing
something new for specialized purpose.

> REST API to list all deployed services by the same user
> -------------------------------------------------------
>
>                 Key: YARN-7215
>                 URL: https://issues.apache.org/jira/browse/YARN-7215
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: api, applications
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> In Slider, it is possible to list deployed applications from the same user by using:
> {code}
> slider list
> {code}
> This API can help UI to display application and services deployed by the same user.
> Apiserver does not have ability to list all applications/services at this time.  This
API requires fast response to list all applications because it is a common UI operation. 
ApiServer deployed applications persist configuration in HDFS similar to slider, but using
directory listing to display deployed application might cost too much overhead to namenode.
 We may want to use alternative storage mechanism to cache deployed application configuration
to accelerate the response time of list deployed applications.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message