Return-Path: X-Original-To: apmail-hadoop-yarn-commits-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1F5F411D5F for ; Sat, 12 Jul 2014 02:26:17 +0000 (UTC) Received: (qmail 52725 invoked by uid 500); 12 Jul 2014 02:26:17 -0000 Delivered-To: apmail-hadoop-yarn-commits-archive@hadoop.apache.org Received: (qmail 52682 invoked by uid 500); 12 Jul 2014 02:26:16 -0000 Mailing-List: contact yarn-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-commits@hadoop.apache.org Delivered-To: mailing list yarn-commits@hadoop.apache.org Received: (qmail 52667 invoked by uid 99); 12 Jul 2014 02:26:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jul 2014 02:26:16 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,WEIRD_PORT X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jul 2014 02:26:12 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 6B6572388CA6; Sat, 12 Jul 2014 02:25:11 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1609878 [7/7] - in /hadoop/common/branches/YARN-1051/hadoop-yarn-project: ./ hadoop-yarn/bin/ hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/ex... Date: Sat, 12 Jul 2014 02:25:04 -0000 To: yarn-commits@hadoop.apache.org From: subru@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20140712022511.6B6572388CA6@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Modified: hadoop/common/branches/YARN-1051/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm URL: http://svn.apache.org/viewvc/hadoop/common/branches/YARN-1051/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm?rev=1609878&r1=1609877&r2=1609878&view=diff ============================================================================== --- hadoop/common/branches/YARN-1051/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm (original) +++ hadoop/common/branches/YARN-1051/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm Sat Jul 12 02:24:40 2014 @@ -195,6 +195,14 @@ ResourceManager REST API's. *---------------+--------------+-------------------------------+ | totalMB | long | The amount of total memory in MB | *---------------+--------------+-------------------------------+ +| reservedVirtualCores | long | The number of reserved virtual cores | +*---------------+--------------+-------------------------------+ +| availableVirtualCores | long | The number of available virtual cores | +*---------------+--------------+-------------------------------+ +| allocatedVirtualCores | long | The number of allocated virtual cores | +*---------------+--------------+-------------------------------+ +| totalVirtualCores | long | The total number of virtual cores | +*---------------+--------------+-------------------------------+ | containersAllocated | int | The number of containers allocated | *---------------+--------------+-------------------------------+ | containersReserved | int | The number of containers reserved | @@ -247,10 +255,14 @@ ResourceManager REST API's. "reservedMB":0, "availableMB":17408, "allocatedMB":0, + "reservedVirtualCores":0, + "availableVirtualCores":7, + "allocatedVirtualCores":1, "containersAllocated":0, "containersReserved":0, "containersPending":0, "totalMB":17408, + "totalVirtualCores":8, "totalNodes":1, "lostNodes":0, "unhealthyNodes":0, @@ -293,10 +305,14 @@ ResourceManager REST API's. 0 17408 0 + 0 + 7 + 1 0 0 0 17408 + 8 1 0 0 @@ -1089,9 +1105,9 @@ ResourceManager REST API's. +---+ -* Cluster Applications API +* {Cluster Applications API} - With the Applications API, you can obtain a collection of resources, each of which represents an application. When you run a GET operation on this resource, you obtain a collection of Application Objects. + With the Applications API, you can obtain a collection of resources, each of which represents an application. When you run a GET operation on this resource, you obtain a collection of Application Objects. ** URI @@ -1107,7 +1123,7 @@ ResourceManager REST API's. ** Query Parameters Supported - Multiple parameters can be specified. The started and finished times have a begin and end parameter to allow you to specify ranges. For example, one could request all applications that started between 1:00am and 2:00pm on 12/19/2011 with startedTimeBegin=1324256400&startedTimeEnd=1324303200. If the Begin parameter is not specified, it defaults to 0, and if the End parameter is not specified, it defaults to infinity. + Multiple parameters can be specified for GET operations. The started and finished times have a begin and end parameter to allow you to specify ranges. For example, one could request all applications that started between 1:00am and 2:00pm on 12/19/2011 with startedTimeBegin=1324256400&startedTimeEnd=1324303200. If the Begin parameter is not specified, it defaults to 0, and if the End parameter is not specified, it defaults to infinity. ------ * state [deprecated] - state of the application @@ -1279,7 +1295,6 @@ _01_000001 0 - +---+ * Cluster Application Statistics API @@ -1411,7 +1426,7 @@ _01_000001 ** URI - Use the following URI to obtain an app object, from a application identified by the {appid} value. + Use the following URI to obtain an app object, from a application identified by the {appid} value. ------ * http:///ws/v1/cluster/apps/{appid} @@ -1522,7 +1537,7 @@ _01_000001 } +---+ - <> + <> HTTP Request: @@ -1564,23 +1579,20 @@ _01_000001 +---+ -* Cluster Application State API - - With the application state API, you can query the state of a submitted app as well kill a running app by modifying the state of a running app using a PUT request with the state set to "KILLED". To perform the PUT operation, authentication has to be setup for the RM web services. In addition, you must be authorized to kill the app. Currently you can only change the state to "KILLED"; an attempt to change the state to any other results in a 400 error response. Examples of the unauthorized and bad request errors are below. When you carry out a successful PUT, the iniital response may be a 202. You can confirm that the app is killed by repeating the PUT request until you get a 200, querying the state using the GET method or querying for app information and checking the state. In the examples below, we repeat the PUT request and get a 200 response. +* Cluster Application Attempts API - Please note that in order to kill an app, you must have an authentication filter setup for the HTTP interface. The functionality requires that a username is set in the HttpServletRequest. If no filter is setup, the response will be an "UNAUTHORIZED" response. + With the application attempts API, you can obtain a collection of resources that represent an application attempt. When you run a GET operation on this resource, you obtain a collection of App Attempt Objects. ** URI ------ - * http:///ws/v1/cluster/apps/{appid}/state ------ +------ + * http:///ws/v1/cluster/apps/{appid}/appattempts +------ -** HTTP Operations Supported +** HTTP Operations Supported ------ * GET - * PUT ------ ** Query Parameters Supported @@ -1589,312 +1601,447 @@ _01_000001 None ------ -** Elements of object +** Elements of the object - When you make a request for the state of an app, the information returned has the following fields + When you make a request for the list of app attempts, the information will be returned as an array of app attempt objects. + + appAttempts: *---------------+--------------+-------------------------------+ || Item || Data Type || Description | *---------------+--------------+-------------------------------+ -| state | string | The application state - can be one of "NEW", "NEW_SAVING", "SUBMITTED", "ACCEPTED", "RUNNING", "FINISHED", "FAILED", "KILLED" | +| appAttempt | array of app attempt objects(JSON)/zero or more app attempt objects(XML) | The collection of app attempt objects | *---------------+--------------+--------------------------------+ +** Elements of the object + +*---------------+--------------+-------------------------------+ +|| Item || Data Type || Description | +*---------------+--------------+-------------------------------+ +| id | string | The app attempt id | +*---------------+--------------+--------------------------------+ +| nodeId | string | The node id of the node the attempt ran on| +*---------------+--------------+--------------------------------+ +| nodeHttpAddress | string | The node http address of the node the attempt ran on| +*---------------+--------------+--------------------------------+ +| logsLink | string | The http link to the app attempt logs | +*---------------+--------------+--------------------------------+ +| containerId | string | The id of the container for the app attempt | +*---------------+--------------+--------------------------------+ +| startTime | long | The start time of the attempt (in ms since epoch)| +*---------------+--------------+--------------------------------+ ** Response Examples - <> + <> - HTTP Request + HTTP Request: ------ - GET http:///ws/v1/cluster/apps/application_1399397633663_0003/state ------ +------ + GET http:///ws/v1/cluster/apps/application_1326821518301_0005/appattempts +------ Response Header: +---+ -HTTP/1.1 200 OK -Content-Type: application/json -Transfer-Encoding: chunked -Server: Jetty(6.1.26) + HTTP/1.1 200 OK + Content-Type: application/json + Transfer-Encoding: chunked + Server: Jetty(6.1.26) +---+ Response Body: +---+ { - "state":"ACCEPTED" + "appAttempts" : { + "appAttempt" : [ + { + "nodeId" : "host.domain.com:8041", + "nodeHttpAddress" : "host.domain.com:8042", + "startTime" : 1326381444693, + "id" : 1, + "logsLink" : "http://host.domain.com:8042/node/containerlogs/container_1326821518301_0005_01_000001/user1", + "containerId" : "container_1326821518301_0005_01_000001" + } + ] + } } +---+ - HTTP Request - ------ - PUT http:///ws/v1/cluster/apps/application_1399397633663_0003/state ----- + <> - Request Body: + HTTP Request: -+---+ -{ - "state":"KILLED" -} -+---+ +------ + GET http:///ws/v1/cluster/apps/application_1326821518301_0005/appattempts + Accept: application/xml +------ Response Header: +---+ -HTTP/1.1 202 Accepted -Content-Type: application/json -Transfer-Encoding: chunked -Location: http:///ws/v1/cluster/apps/application_1399397633663_0003 -Server: Jetty(6.1.26) + HTTP/1.1 200 OK + Content-Type: application/xml + Content-Length: 575 + Server: Jetty(6.1.26) +---+ Response Body: +---+ -{ - "state":"ACCEPTED" -} + + + + host.domain.com:8042 + host.domain.com:8041 + 1 + 1326381444693 + container_1326821518301_0005_01_000001 + http://host.domain.com:8042/node/containerlogs/container_1326821518301_0005_01_000001/user1 + + +---+ ------ - PUT http:///ws/v1/cluster/apps/application_1399397633663_0003/state ----- +* Cluster Nodes API - Request Body: + With the Nodes API, you can obtain a collection of resources, each of which represents a node. When you run a GET operation on this resource, you obtain a collection of Node Objects. -+---+ -{ - "state":"KILLED" -} -+---+ +** URI - Response Header: +------ + * http:///ws/v1/cluster/nodes +------ -+---+ -HTTP/1.1 200 OK -Content-Type: application/json -Transfer-Encoding: chunked -Server: Jetty(6.1.26) -+---+ +** HTTP Operations Supported - Response Body: +------ + * GET +------ -+---+ -{ - "state":"KILLED" -} -+---+ +** Query Parameters Supported - <> +------ + * state - the state of the node + * healthy - true or false +------ - HTTP Request +** Elements of the object ------ - GET http:///ws/v1/cluster/apps/application_1399397633663_0003/state ------ + When you make a request for the list of nodes, the information will be returned as a collection of node objects. + See also {{Node API}} for syntax of the node object. + +*---------------+--------------+-------------------------------+ +|| Item || Data Type || Description | +*---------------+--------------+-------------------------------+ +| node | array of node objects(JSON)/zero or more node objects(XML) | A collection of node objects | +*---------------+--------------+-------------------------------+ + +** Response Examples + + <> + + HTTP Request: + +------ + GET http:///ws/v1/cluster/nodes +------ Response Header: +---+ -HTTP/1.1 200 OK -Content-Type: application/xml -Content-Length: 99 -Server: Jetty(6.1.26) + HTTP/1.1 200 OK + Content-Type: application/json + Transfer-Encoding: chunked + Server: Jetty(6.1.26) +---+ Response Body: +---+ - - - ACCEPTED - +{ + "nodes": + { + "node": + [ + { + "rack":"\/default-rack", + "state":"NEW", + "id":"h2:1235", + "nodeHostName":"h2", + "nodeHTTPAddress":"h2:2", + "healthStatus":"Healthy", + "lastHealthUpdate":1324056895432, + "healthReport":"Healthy", + "numContainers":0, + "usedMemoryMB":0, + "availMemoryMB":8192, + "usedVirtualCores":0, + "availableVirtualCores":8 + }, + { + "rack":"\/default-rack", + "state":"NEW", + "id":"h1:1234", + "nodeHostName":"h1", + "nodeHTTPAddress":"h1:2", + "healthStatus":"Healthy", + "lastHealthUpdate":1324056895092, + "healthReport":"Healthy", + "numContainers":0, + "usedMemoryMB":0, + "availMemoryMB":8192, + "usedVirtualCores":0, + "availableVirtualCores":8 + } + ] + } +} +---+ - HTTP Request - ------ - PUT http:///ws/v1/cluster/apps/application_1399397633663_0003/state ----- + <> - Request Body: + HTTP Request: -+---+ - - - KILLED - -+---+ +------ + GET http:///ws/v1/cluster/nodes + Accept: application/xml +------ Response Header: +---+ -HTTP/1.1 202 Accepted -Content-Type: application/json -Content-Length: 794 -Location: http:///ws/v1/cluster/apps/application_1399397633663_0003 -Server: Jetty(6.1.26) + HTTP/1.1 200 OK + Content-Type: application/xml + Content-Length: 1104 + Server: Jetty(6.1.26) +---+ Response Body: +---+ - - ACCEPTED - -+---+ - - HTTP Request + + + /default-rack + RUNNING + h2:1234 + h2 + h2:2 + Healthy + 1324333268447 + Healthy + 0 + 0 + 5120 + 0 + 8 + + + /default-rack + RUNNING + h1:1234 + h1 + h1:2 + Healthy + 1324333268447 + Healthy + 0 + 0 + 5120 + 0 + 8 + + ++---+ ------ - PUT http:///ws/v1/cluster/apps/application_1399397633663_0003/state ----- - Request Body: +* Cluster {Node API} -+---+ - - - KILLED - -+---+ + A node resource contains information about a node in the cluster. - Response Header: +** URI -+---+ -HTTP/1.1 200 OK -Content-Type: application/xml -Content-Length: 917 -Server: Jetty(6.1.26) -+---+ + Use the following URI to obtain a Node Object, from a node identified by the {nodeid} value. - Response Body: +------ + * http:///ws/v1/cluster/nodes/{nodeid} +------ -+---+ - - - KILLED - -+---+ +** HTTP Operations Supported - <> +------ + * GET +------ - HTTP Request +** Query Parameters Supported ------ - PUT http:///ws/v1/cluster/apps/application_1399397633663_0003/state ----- +------ + None +------ - Request Body: +** Elements of the object -+---+ - - - KILLED - -+---+ +*---------------+--------------+-------------------------------+ +|| Item || Data Type || Description | +*---------------+--------------+-------------------------------+ +| rack | string | The rack location of this node | +*---------------+--------------+-------------------------------+ +| state | string | State of the node - valid values are: NEW, RUNNING, UNHEALTHY, DECOMMISSIONED, LOST, REBOOTED | +*---------------+--------------+-------------------------------+ +| id | string | The node id | +*---------------+--------------+-------------------------------+ +| nodeHostName | string | The host name of the node| +*---------------+--------------+-------------------------------+ +| nodeHTTPAddress | string | The nodes HTTP address| +*---------------+--------------+-------------------------------+ +| healthStatus | string | The health status of the node - Healthy or Unhealthy | +*---------------+--------------+-------------------------------+ +| healthReport | string | A detailed health report | +*---------------+--------------+-------------------------------+ +| lastHealthUpdate | long | The last time the node reported its health (in ms since epoch)| +*---------------+--------------+-------------------------------+ +| usedMemoryMB | long | The total amount of memory currently used on the node (in MB)| +*---------------+--------------+-------------------------------+ +| availMemoryMB | long | The total amount of memory currently available on the node (in MB)| +*---------------+--------------+-------------------------------+ +| usedVirtualCores | long | The total number of vCores currently used on the node | +*---------------+--------------+-------------------------------+ +| availableVirtualCores | long | The total number of vCores available on the node | +*---------------+--------------+-------------------------------+ +| numContainers | int | The total number of containers currently running on the node| +*---------------+--------------+-------------------------------+ - Response Header: +** Response Examples -+---+ -HTTP/1.1 403 Unauthorized -Content-Type: application/json -Transfer-Encoding: chunked -Server: Jetty(6.1.26) -+---+ + <> + HTTP Request: - <> +------ + GET http:///ws/v1/cluster/nodes/h2:1235 +------ - HTTP Request + Response Header: ------ - PUT http:///ws/v1/cluster/apps/application_1399397633663_0003/state ----- ++---+ + HTTP/1.1 200 OK + Content-Type: application/json + Transfer-Encoding: chunked + Server: Jetty(6.1.26) ++---+ - Request Body: + Response Body: +---+ - - - RUNNING - +{ + "node": + { + "rack":"\/default-rack", + "state":"NEW", + "id":"h2:1235", + "nodeHostName":"h2", + "nodeHTTPAddress":"h2:2", + "healthStatus":"Healthy", + "lastHealthUpdate":1324056895432, + "healthReport":"Healthy", + "numContainers":0, + "usedMemoryMB":0, + "availMemoryMB":5120, + "usedVirtualCores":0, + "availableVirtualCores":8 + } +} +---+ + <> + + HTTP Request: + +------ + GET http:///ws/v1/cluster/node/h2:1235 + Accept: application/xml +------ + Response Header: +---+ -HTTP/1.1 400 -Content-Length: 295 -Content-Type: application/xml -Server: Jetty(6.1.26) + HTTP/1.1 200 OK + Content-Type: application/xml + Content-Length: 552 + Server: Jetty(6.1.26) +---+ Response Body: +---+ - - BadRequestException - java.lang.Exception: Only 'KILLED' is allowed as a target state. - org.apache.hadoop.yarn.webapp.BadRequestException - + + /default-rack + NEW + h2:1235 + h2 + h2:2 + Healthy + 1324333268447 + Healthy + 0 + 0 + 5120 + 0 + 5120 + +---+ -* Cluster Application Attempts API +* {Cluster Writeable APIs} - With the application attempts API, you can obtain a collection of resources that represent an application attempt. When you run a GET operation on this resource, you obtain a collection of App Attempt Objects. + The setions below refer to APIs which allow to create and modify applications. These APIs are currently in alpha and may change in the future. + +* {Cluster New Application API} + + With the New Application API, you can obtain an application-id which can then be used as part of the {{{Cluster_Applications_API(Submit_Application)}Cluster Submit Applications API}} to submit applications. The response also includes the maximum resource capabilities available on the cluster. + + This feature is currently in the alpha stage and may change in the future. ** URI ------ - * http:///ws/v1/cluster/apps/{appid}/appattempts + * http:///ws/v1/cluster/apps/new-application ------ -** HTTP Operations Supported +** HTTP Operations Supported ------ - * GET + * POST ------ ** Query Parameters Supported ------ - None + * None ------ -** Elements of the object - - When you make a request for the list of app attempts, the information will be returned as an array of app attempt objects. +** Elements of the NewApplication object - appAttempts: + The NewApplication response contains the following elements: *---------------+--------------+-------------------------------+ || Item || Data Type || Description | *---------------+--------------+-------------------------------+ -| appAttempt | array of app attempt objects(JSON)/zero or more app attempt objects(XML) | The collection of app attempt objects | +| application-id | string | The newly created application id | +*---------------+--------------+--------------------------------+ +| maximum-resource-capabilities | object | The maximum resource capabilities available on this cluster | *---------------+--------------+--------------------------------+ -** Elements of the object + The object contains the following elements: + *---------------+--------------+-------------------------------+ || Item || Data Type || Description | *---------------+--------------+-------------------------------+ -| id | string | The app attempt id | -*---------------+--------------+--------------------------------+ -| nodeId | string | The node id of the node the attempt ran on| -*---------------+--------------+--------------------------------+ -| nodeHttpAddress | string | The node http address of the node the attempt ran on| -*---------------+--------------+--------------------------------+ -| logsLink | string | The http link to the app attempt logs | -*---------------+--------------+--------------------------------+ -| containerId | string | The id of the container for the app attempt | +| memory | int | The maxiumim memory available for a container | *---------------+--------------+--------------------------------+ -| startTime | long | The start time of the attempt (in ms since epoch)| +| vCores | int | The maximum number of cores available for a container | *---------------+--------------+--------------------------------+ ** Response Examples @@ -1904,7 +2051,7 @@ Server: Jetty(6.1.26) HTTP Request: ------ - GET http:///ws/v1/cluster/apps/application_1326821518301_0005/appattempts + POST http:///ws/v1/cluster/apps/new-application ------ Response Header: @@ -1920,17 +2067,11 @@ Server: Jetty(6.1.26) +---+ { - "appAttempts" : { - "appAttempt" : [ - { - "nodeId" : "host.domain.com:8041", - "nodeHttpAddress" : "host.domain.com:8042", - "startTime" : 1326381444693, - "id" : 1, - "logsLink" : "http://host.domain.com:8042/node/containerlogs/container_1326821518301_0005_01_000001/user1", - "containerId" : "container_1326821518301_0005_01_000001" - } - ] + "application-id":"application_1404198295326_0001", + "maximum-resource-capability": + { + "memory":"8192", + "vCores":"32" } } +---+ @@ -1940,8 +2081,7 @@ Server: Jetty(6.1.26) HTTP Request: ------ - GET http:///ws/v1/cluster/apps/application_1326821518301_0005/appattempts - Accept: application/xml + POST http:///ws/v1/cluster/apps/new-application ------ Response Header: @@ -1949,7 +2089,7 @@ Server: Jetty(6.1.26) +---+ HTTP/1.1 200 OK Content-Type: application/xml - Content-Length: 575 + Content-Length: 248 Server: Jetty(6.1.26) +---+ @@ -1957,179 +2097,351 @@ Server: Jetty(6.1.26) +---+ - - - host.domain.com:8042 - host.domain.com:8041 - 1 - 1326381444693 - container_1326821518301_0005_01_000001 - http://host.domain.com:8042/node/containerlogs/container_1326821518301_0005_01_000001/user1 - - + + application_1404198295326_0003 + + 8192 + 32 + + +---+ -* Cluster Nodes API +* {Cluster Applications API(Submit Application)} - With the Nodes API, you can obtain a collection of resources, each of which represents a node. When you run a GET operation on this resource, you obtain a collection of Node Objects. + The Submit Applications API can be used to submit applications. In case of submitting applications, you must first obtain an application-id using the {{{Cluster_New_Application_API}Cluster New Application API}}. The application-id must be part of the request body. The response contains a URL to the application page which can be used to track the state and progress of your application. ** URI ------ - * http:///ws/v1/cluster/nodes + * http:///ws/v1/cluster/apps ------ ** HTTP Operations Supported ------ - * GET + * POST ------ -** Query Parameters Supported +** POST Response Examples ------- - * state - the state of the node - * healthy - true or false ------- + POST requests can be used to submit apps to the ResourceManager. As mentioned above, an application-id must be obtained first. Successful submissions result in a 202 response code and a Location header specifying where to get information about the app. Please note that in order to submit an app, you must have an authentication filter setup for the HTTP interface. The functionality requires that a username is set in the HttpServletRequest. If no filter is setup, the response will be an "UNAUTHORIZED" response. -** Elements of the object + Please note that this feature is currently in the alpha stage and may change in the future. - When you make a request for the list of nodes, the information will be returned as a collection of node objects. - See also {{Node API}} for syntax of the node object. +*** Elements of the POST request object *---------------+--------------+-------------------------------+ -|| Item || Data Type || Description | +|| Item || Data Type || Description | *---------------+--------------+-------------------------------+ -| node | array of node objects(JSON)/zero or more node objects(XML) | A collection of node objects | +| application-id | string | The application id | +*---------------+--------------+-------------------------------+ +| application-name | string | The application name | +*---------------+--------------+-------------------------------+ +| queue | string | The name of the queue to which the application should be submitted | +*---------------+--------------+-------------------------------+ +| priority | int | The priority of the application | +*---------------+--------------+-------------------------------+ +| am-container-spec | object | The application master container launch context, described below | +*---------------+--------------+-------------------------------+ +| unmanaged-AM | boolean | Is the application using an unmanaged application master | +*---------------+--------------+-------------------------------+ +| max-app-attempts | int | The max number of attempts for this application | +*---------------+--------------+-------------------------------+ +| resource | object | The resources the application master requires, described below | +*---------------+--------------+-------------------------------+ +| application-type | string | The application type(MapReduce, Pig, Hive, etc) | +*---------------+--------------+-------------------------------+ +| keep-containers-across-application-attempts | boolean | Should YARN keep the containers used by this application instead of destroying them | +*---------------+--------------+-------------------------------+ +| application-tags | object | List of application tags, please see the request examples on how to speciy the tags | *---------------+--------------+-------------------------------+ -** Response Examples + Elements of the object + + The am-container-spec object should be used to provide the container launch context for the application master. + +*---------------+--------------+-------------------------------+ +|| Item || Data Type || Description | +*---------------+--------------+-------------------------------+ +| local-resources | object | Object describing the resources that need to be localized, described below | +*---------------+--------------+-------------------------------+ +| environment | object | Environment variables for your containers, specified as key value pairs | +*---------------+--------------+-------------------------------+ +| commands | object | The commands for launching your container, in the order in which they should be executed | +*---------------+--------------+-------------------------------+ +| service-data | object | Application specific service data; key is the name of the auxiliary servce, value is base-64 encoding of the data you wish to pass | +*---------------+--------------+-------------------------------+ +| credentials | object | The credentials required for your application to run, described below | +*---------------+--------------+-------------------------------+ +| application-acls | objec | ACLs for your application; the key can be "VIEW_APP" or "MODIFY_APP", the value is the list of users with the permissions | +*---------------+--------------+-------------------------------+ + + Elements of the object + + The object is a collection of key-value pairs. They key is an identifier for the resources to be localized and the value is the details of the resource. The elements of the value are described below: + +*---------------+--------------+-------------------------------+ +|| Item || Data Type || Description | +*---------------+--------------+-------------------------------+ +| resource | string | Location of the resource to be localized | +*---------------+--------------+-------------------------------+ +| type | string | Type of the resource; options are "ARCHIVE", "FILE", and "PATTERN" | +*---------------+--------------+-------------------------------+ +| visibility | string | Visibility the resource to be localized; options are "PUBLIC", "PRIVATE", and "APPLICATION" | +*---------------+--------------+-------------------------------+ +| size | long | Size of the resource to be localized | +*---------------+--------------+-------------------------------+ +| timestamp | long | Timestamp of the resource to be localized | +*---------------+--------------+-------------------------------+ + + Elements of the object + + The credentials object should be used to pass data required for the application to authenticate itself such as delegation-tokens and secrets. + +*---------------+--------------+-------------------------------+ +|| Item || Data Type || Description | +*---------------+--------------+-------------------------------+ +| tokens | object | Tokens that you wish to pass to your application, specified as key-value pairs. The key is an identifier for the token and the value is the token(which should be obtained using the respective web-services) | +*---------------+--------------+-------------------------------+ +| secrets | object | Secrets that you wish to use in your application, specified as key-value pairs. They key is an identifier and the value is the base-64 encoding of the secret | +*---------------+--------------+-------------------------------+ + + + Elements of the POST request body object + +*---------------+--------------+-------------------------------+ +|| Item || Data Type || Description | +*---------------+--------------+-------------------------------+ +| memory | int | Memory required for each container | +*---------------+--------------+-------------------------------+ +| vCores | int | Virtual cores required for each container | +*---------------+--------------+-------------------------------+ <> HTTP Request: ------- - GET http:///ws/v1/cluster/nodes ------- ++---+ + POST http:///ws/v1/cluster/apps + Accept: application/json + Content-Type: application/json + { + "application-id":"application_1404203615263_0001", + "application-name":"test", + "queue":"testqueue", + "priority":"3", + "am-container-spec": + { + "local-resources": + { + "entry": + { + "key":"example", + "value": + { + "resource":"http://www.test.com/file.txt", + "type":"FILE", + "visibility":"APPLICATION", + "size":"100", + "timestamp":"1404203616003" + } + } + }, + "environment": + { + "entry": + { + "key":"APP_VAR", + "value":"ENV_SETTING" + } + }, + "commands": + { + "command":"/bin/sleep 5" + }, + "service-data": + { + "entry": + { + "key":"test", + "value":"dmFsdWUxMg" + } + }, + "credentials": + { + "tokens":null, + "secrets": + { + "entry": + { + "key":"secret1", + "value":"c2VjcmV0MQ" + } + } + }, + "application-acls": + { + "entry": + [ + { + "key":"VIEW_APP", + "value":"testuser3, testuser4" + }, + { + "key":"MODIFY_APP", + "value":"testuser1, testuser2" + } + ] + } + }, + "unmanaged-AM":"false", + "max-app-attempts":"2", + "resource": + { + "memory":"1024", + "vCores":"1" + }, + "application-type":"YARN", + "keep-containers-across-application-attempts":"false", + "application-tags": + { + "tag": + [ + "tag 2", + "tag1" + ] + } + } ++---+ Response Header: +---+ - HTTP/1.1 200 OK - Content-Type: application/json + HTTP/1.1 202 Transfer-Encoding: chunked + Location: http:///ws/v1/cluster/apps/application_1404203615263_0001 + Content-Type: application/json Server: Jetty(6.1.26) +---+ Response Body: +---+ -{ - "nodes": - { - "node": - [ - { - "rack":"\/default-rack", - "state":"NEW", - "id":"h2:1235", - "nodeHostName":"h2", - "nodeHTTPAddress":"h2:2", - "healthStatus":"Healthy", - "lastHealthUpdate":1324056895432, - "healthReport":"Healthy", - "numContainers":0, - "usedMemoryMB":0 - "availMemoryMB":8192 - }, - { - "rack":"\/default-rack", - "state":"NEW", - "id":"h1:1234", - "nodeHostName":"h1", - "nodeHTTPAddress":"h1:2", - "healthStatus":"Healthy", - "lastHealthUpdate":1324056895092, - "healthReport":"Healthy", - "numContainers":0, - "usedMemoryMB":0, - "availMemoryMB":8192 - } - ] - } -} + No response body +---+ - <> + <> HTTP Request: ------- - GET http:///ws/v1/cluster/nodes ++---+ + POST http:///ws/v1/cluster/apps Accept: application/xml ------- + Content-Type: application/xml + + + application_1404204891930_0002 + test + testqueue + 3 + + + + example + + http://www.test.com/file.txt + FILE + APPLICATION + 100 + 1404204892877 + + + + + + APP_VAR + ENV_SETTING + + + + /bin/sleep 5 + + + + test + dmFsdWUxMg + + + + + + + secret1 + c2VjcmV0MQ + + + + + + VIEW_APP + testuser3, testuser4 + + + MODIFY_APP + testuser1, testuser2 + + + + false + 2 + + 1024 + 1 + + YARN + false + + tag 2 + tag1 + + ++---+ Response Header: +---+ - HTTP/1.1 200 OK + HTTP/1.1 202 + Transfer-Encoding: chunked + Location: http:///ws/v1/cluster/apps/application_1404204891930_0002 Content-Type: application/xml - Content-Length: 1104 Server: Jetty(6.1.26) +---+ Response Body: +---+ - - - - /default-rack - RUNNING - h2:1234 - h2 - h2:2 - Healthy - 1324333268447 - Healthy - 0 - 0 - 5120 - - - /default-rack - RUNNING - h1:1234 - h1 - h1:2 - Healthy - 1324333268447 - Healthy - 0 - 0 - 5120 - - + No response body +---+ +* Cluster Application State API -* Cluster {Node API} + With the application state API, you can query the state of a submitted app as well kill a running app by modifying the state of a running app using a PUT request with the state set to "KILLED". To perform the PUT operation, authentication has to be setup for the RM web services. In addition, you must be authorized to kill the app. Currently you can only change the state to "KILLED"; an attempt to change the state to any other results in a 400 error response. Examples of the unauthorized and bad request errors are below. When you carry out a successful PUT, the iniital response may be a 202. You can confirm that the app is killed by repeating the PUT request until you get a 200, querying the state using the GET method or querying for app information and checking the state. In the examples below, we repeat the PUT request and get a 200 response. - A node resource contains information about a node in the cluster. + Please note that in order to kill an app, you must have an authentication filter setup for the HTTP interface. The functionality requires that a username is set in the HttpServletRequest. If no filter is setup, the response will be an "UNAUTHORIZED" response. -** URI + This feature is currently in the alpha stage and may change in the future. - Use the following URI to obtain a Node Object, from a node identified by the {nodeid} value. +** URI ------- - * http:///ws/v1/cluster/nodes/{nodeid} ------- +----- + * http:///ws/v1/cluster/apps/{appid}/state +----- -** HTTP Operations Supported +** HTTP Operations Supported ------ * GET + * PUT ------ ** Query Parameters Supported @@ -2138,108 +2450,260 @@ Server: Jetty(6.1.26) None ------ -** Elements of the object +** Elements of object + + When you make a request for the state of an app, the information returned has the following fields *---------------+--------------+-------------------------------+ -|| Item || Data Type || Description | -*---------------+--------------+-------------------------------+ -| rack | string | The rack location of this node | -*---------------+--------------+-------------------------------+ -| state | string | State of the node - valid values are: NEW, RUNNING, UNHEALTHY, DECOMMISSIONED, LOST, REBOOTED | -*---------------+--------------+-------------------------------+ -| id | string | The node id | -*---------------+--------------+-------------------------------+ -| nodeHostName | string | The host name of the node| -*---------------+--------------+-------------------------------+ -| nodeHTTPAddress | string | The nodes HTTP address| -*---------------+--------------+-------------------------------+ -| healthStatus | string | The health status of the node - Healthy or Unhealthy | -*---------------+--------------+-------------------------------+ -| healthReport | string | A detailed health report | -*---------------+--------------+-------------------------------+ -| lastHealthUpdate | long | The last time the node reported its health (in ms since epoch)| -*---------------+--------------+-------------------------------+ -| usedMemoryMB | long | The total about of memory currently used on the node (in MB)| -*---------------+--------------+-------------------------------+ -| availMemoryMB | long | The total amount of memory currently available on the node (in MB)| -*---------------+--------------+-------------------------------+ -| numContainers | int | The total number of containers currently running on the node| +|| Item || Data Type || Description | *---------------+--------------+-------------------------------+ +| state | string | The application state - can be one of "NEW", "NEW_SAVING", "SUBMITTED", "ACCEPTED", "RUNNING", "FINISHED", "FAILED", "KILLED" | +*---------------+--------------+--------------------------------+ + ** Response Examples - <> + <> - HTTP Request: + HTTP Request ------- - GET http:///ws/v1/cluster/nodes/h2:1235 ------- +----- + GET http:///ws/v1/cluster/apps/application_1399397633663_0003/state +----- Response Header: +---+ - HTTP/1.1 200 OK - Content-Type: application/json - Transfer-Encoding: chunked - Server: Jetty(6.1.26) +HTTP/1.1 200 OK +Content-Type: application/json +Transfer-Encoding: chunked +Server: Jetty(6.1.26) +---+ Response Body: +---+ { - "node": - { - "rack":"\/default-rack", - "state":"NEW", - "id":"h2:1235", - "nodeHostName":"h2", - "nodeHTTPAddress":"h2:2", - "healthStatus":"Healthy", - "lastHealthUpdate":1324056895432, - "healthReport":"Healthy", - "numContainers":0, - "usedMemoryMB":0, - "availMemoryMB":5120 - } + "state":"ACCEPTED" } +---+ - <> + HTTP Request - HTTP Request: +----- + PUT http:///ws/v1/cluster/apps/application_1399397633663_0003/state +---- ------- - GET http:///ws/v1/cluster/node/h2:1235 - Accept: application/xml ------- + Request Body: + ++---+ +{ + "state":"KILLED" +} ++---+ Response Header: +---+ - HTTP/1.1 200 OK - Content-Type: application/xml - Content-Length: 552 - Server: Jetty(6.1.26) +HTTP/1.1 202 Accepted +Content-Type: application/json +Transfer-Encoding: chunked +Location: http:///ws/v1/cluster/apps/application_1399397633663_0003 +Server: Jetty(6.1.26) ++---+ + + Response Body: + ++---+ +{ + "state":"ACCEPTED" +} ++---+ + +----- + PUT http:///ws/v1/cluster/apps/application_1399397633663_0003/state +---- + + Request Body: + ++---+ +{ + "state":"KILLED" +} ++---+ + + Response Header: + ++---+ +HTTP/1.1 200 OK +Content-Type: application/json +Transfer-Encoding: chunked +Server: Jetty(6.1.26) ++---+ + + Response Body: + ++---+ +{ + "state":"KILLED" +} ++---+ + + <> + + HTTP Request + +----- + GET http:///ws/v1/cluster/apps/application_1399397633663_0003/state +----- + + Response Header: + ++---+ +HTTP/1.1 200 OK +Content-Type: application/xml +Content-Length: 99 +Server: Jetty(6.1.26) +---+ Response Body: +---+ - - /default-rack - NEW - h2:1235 - h2 - h2:2 - Healthy - 1324333268447 - Healthy - 0 - 0 - 5120 - + + ACCEPTED + ++---+ + + HTTP Request + +----- + PUT http:///ws/v1/cluster/apps/application_1399397633663_0003/state +---- + + Request Body: + ++---+ + + + KILLED + ++---+ + + Response Header: + ++---+ +HTTP/1.1 202 Accepted +Content-Type: application/json +Content-Length: 794 +Location: http:///ws/v1/cluster/apps/application_1399397633663_0003 +Server: Jetty(6.1.26) ++---+ + + Response Body: + ++---+ + + + ACCEPTED + ++---+ + + HTTP Request + +----- + PUT http:///ws/v1/cluster/apps/application_1399397633663_0003/state +---- + + Request Body: + ++---+ + + + KILLED + ++---+ + + Response Header: + +---+ +HTTP/1.1 200 OK +Content-Type: application/xml +Content-Length: 917 +Server: Jetty(6.1.26) ++---+ + + Response Body: + ++---+ + + + KILLED + ++---+ + + <> + + HTTP Request + +----- + PUT http:///ws/v1/cluster/apps/application_1399397633663_0003/state +---- + + Request Body: + ++---+ + + + KILLED + ++---+ + + Response Header: + ++---+ +HTTP/1.1 403 Unauthorized +Content-Type: application/json +Transfer-Encoding: chunked +Server: Jetty(6.1.26) ++---+ + + + <> + + HTTP Request + +----- + PUT http:///ws/v1/cluster/apps/application_1399397633663_0003/state +---- + + Request Body: + ++---+ + + + RUNNING + ++---+ + + Response Header: + ++---+ +HTTP/1.1 400 +Content-Length: 295 +Content-Type: application/xml +Server: Jetty(6.1.26) ++---+ + + Response Body: + ++---+ + + + BadRequestException + java.lang.Exception: Only 'KILLED' is allowed as a target state. + org.apache.hadoop.yarn.webapp.BadRequestException + ++---+ + Modified: hadoop/common/branches/YARN-1051/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YarnCommands.apt.vm URL: http://svn.apache.org/viewvc/hadoop/common/branches/YARN-1051/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YarnCommands.apt.vm?rev=1609878&r1=1609877&r2=1609878&view=diff ============================================================================== --- hadoop/common/branches/YARN-1051/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YarnCommands.apt.vm (original) +++ hadoop/common/branches/YARN-1051/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YarnCommands.apt.vm Sat Jul 12 02:24:40 2014 @@ -157,9 +157,17 @@ Usage: yarn [--config confdir] COMMAND Start the ResourceManager ------- - Usage: yarn resourcemanager + Usage: yarn resourcemanager [-format] ------- +*---------------+--------------+ +|| COMMAND_OPTIONS || Description | +*---------------+--------------+ +| -format | Formats the RMStateStore. This will clear the RMStateStore and is +| | useful if past applications are no longer needed. This should be run +| | only when the ResourceManager is not running. +*---------------+--------------+ + ** nodemanager Start the NodeManager