hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7217) Improve API service usability for updating service spec and state
Date Sat, 28 Oct 2017 04:48:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16223235#comment-16223235
] 

Hadoop QA commented on YARN-7217:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 16s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 34 new or modified test files. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 51s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m  4s{color}
| {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 55s{color} |
{color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m  9s{color}
| {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 46s{color} |
{color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 29s{color}
| {color:green} branch has no errors when building and testing our client artifacts. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  0s{color} | {color:blue}
Skipped patched modules with no Java source: hadoop-project {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 26s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in yarn-native-services has 1 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 21s{color} |
{color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 59s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 31s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 54s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 54s{color} | {color:green}
the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  2m 16s{color}
| {color:orange} root: The patch generated 10 new + 247 unchanged - 9 fixed = 257 total (was
256) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 11s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m 14s{color} | {color:green}
The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  9m 53s{color}
| {color:green} patch has no errors when building and testing our client artifacts. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  0s{color} | {color:blue}
Skipped patched modules with no Java source: hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 35s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 26s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 14s{color} | {color:green}
hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 43s{color} | {color:green}
hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  9s{color} | {color:green}
hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 36s{color} | {color:green}
hadoop-yarn-services-core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  9s{color} | {color:green}
hadoop-yarn-services-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 31s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 36s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7217 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12894498/YARN-7217.yarn-native-services.006.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient
 xml  findbugs  checkstyle  |
| uname | Linux 30abcb84a641 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | yarn-native-services / be46757 |
| maven | version: Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00)
|
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/18218/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
|
| checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/18218/artifact/out/diff-checkstyle-root.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/18218/testReport/ |
| modules | C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api U: . |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/18218/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Improve API service usability for updating service spec and state
> -----------------------------------------------------------------
>
>                 Key: YARN-7217
>                 URL: https://issues.apache.org/jira/browse/YARN-7217
>             Project: Hadoop YARN
>          Issue Type: Task
>          Components: api, applications
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: YARN-7217.yarn-native-services.001.patch, YARN-7217.yarn-native-services.002.patch,
YARN-7217.yarn-native-services.003.patch, YARN-7217.yarn-native-services.004.patch, YARN-7217.yarn-native-services.005.patch,
YARN-7217.yarn-native-services.006.patch
>
>
> API service for deploy, and manage YARN services have several limitations.
> {{updateService}} API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.  (This was removed in YARN-7323).
> The overloading is buggy depending on how the configuration should be applied.
> h4. Scenario 1
> A user retrieves Service object from getService call, and the Service object contains
state: STARTED.  The user would like to increase number of containers for the deployed service.
 The JSON has been updated to increase container count.  The PUT method does not actually
increase container count.
> h4. Scenario 2
> A user retrieves Service object from getService call, and the Service object contains
state: STOPPED.  The user would like to make a environment configuration change.  The configuration
does not get updated after PUT method.
> This is possible to address by rearranging the logic of START/STOP after configuration
update.  However, there are other potential combinations that can break PUT method.  For example,
user like to make configuration changes, but not yet restart the service until a later time.
> h4. Scenario 3
> There is no API to list all deployed applications by the same user.
> h4. Scenario 4
> Desired state (spec) and current state are represented by the same Service object.  There
is no easy way to identify "state" is desired state to reach or, the current state of the
service.  It would be nice to have ability to retrieve both desired state, and current state
with separated entry points.  By implementing /spec and /state, it can resolve this problem.
> h4. Scenario 5
> List all services deploy by the same user can trigger a directory listing operation on
namenode if hdfs is used as storage for metadata.  When hundred of users use Service UI to
view or deploy applications, this will trigger denial of services attack on namenode.  The
sparse small metadata files also reduce efficiency of Namenode memory usage.  Hence, a cache
layer for storing service metadata can reduce namenode stress.
> h3. Proposed change
> ApiService can separate the PUT method into two PUT methods for configuration changes
vs operation changes.  New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/spec
> Request Data:
> {
>   "name": "amp",
>   "components": [
>     {
>       "name": "mysql",
>       "number_of_containers": 2,
>       "artifact": {
>         "id": "centos/mysql-57-centos7:latest",
>         "type": "DOCKER"
>       },
>       "run_privileged_container": false,
>       "launch_command": "",
>       "resource": {
>         "cpus": 1,
>         "memory": "2048"
>       },
>       "configuration": {
>         "env": {
>           "MYSQL_USER":"${USER}",
>           "MYSQL_PASSWORD":"password"
>         }
>       }
>      }
>   ],
>   "quicklinks": {
>     "Apache Document Root": "http://httpd.${SERVICE_NAME}.${USER}.${DOMAIN}:8080/",
>     "PHP MyAdmin": "http://phpmyadmin.${SERVICE_NAME}.${USER}.${DOMAIN}:8080/"
>   }
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
>   "name": "amp",
>   "components": [
>     {
>       "name": "mysql",
>       "state": "STOPPED"
>      }
>   ]
> }
> {code}
> SOLR can be used to cache Yarnfile to improve lookup performance and reduce stress of
namenode small file problems and high frequency lookup.  SOLR is chosen for caching metadata
because its indexing feature can be used to build full text search for application catalog
as well.
> For service that requires configuration changes to increase or decrease node count. 
The calling sequence is:
> {code}
> # GET /ws/v1/services/{service_name}/spec
> # Change number_of_containers to desired number.
> # PUT /ws/v1/services/{service_name}/spec to update the spec.
> # PUT /ws/v1/services/{service_name}/state to stop existing service.
> # PUT /ws/v1/services/{service_name}/state to start service.
> {code}
> For components that can increase node count without rewrite configuration:
> {code}
> # GET /ws/v1/services/{service_name}/spec
> # Change number_of_containers to desired number.
> # PUT /ws/v1/services/{service_name}/spec to update the spec.
> # PUT /ws/v1/services/{service_name}/component/{component_name} to change node count.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message