hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gour Saha (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (YARN-8243) Flex down should first remove pending container requests (if any) and then kill running containers
Date Thu, 03 May 2018 22:15:00 GMT

     [ https://issues.apache.org/jira/browse/YARN-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Gour Saha reassigned YARN-8243:
-------------------------------

    Assignee: Gour Saha

> Flex down should first remove pending container requests (if any) and then kill running
containers
> --------------------------------------------------------------------------------------------------
>
>                 Key: YARN-8243
>                 URL: https://issues.apache.org/jira/browse/YARN-8243
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: yarn-native-services
>    Affects Versions: 3.1.0
>            Reporter: Gour Saha
>            Assignee: Gour Saha
>            Priority: Major
>
> This is easy to test on a service with anti-affinity component, to simulate pending container
requests. It can be simulated by other means also (no resource left in cluster, etc.).
> Service yarnfile used to test this -
> {code:java}
> {
>   "name": "sleeper-service",
>   "version": "1",
>   "components" :
>   [
>     {
>       "name": "ping",
>       "number_of_containers": 2,
>       "resource": {
>         "cpus": 1,
>         "memory": "256"
>       },
>       "launch_command": "sleep 9000",
>       "placement_policy": {
>         "constraints": [
>           {
>             "type": "ANTI_AFFINITY",
>             "scope": "NODE",
>             "target_tags": [
>               "ping"
>             ]
>           }
>         ]
>       }
>     }
>   ]
> }
> {code}
> Launch a service with the above yarnfile as below -
> {code:java}
> yarn app -launch simple-aa-1 simple_AA.json
> {code}
> Let's assume there are only 5 nodes in this cluster. Now, flex the above service to 1
extra container than the number of nodes (6 in my case).
> {code:java}
> yarn app -flex simple-aa-1 -component ping 6
> {code}
> Only 5 containers will be allocated and running for simple-aa-1. At this point, flex
it down to 5 containers -
> {code:java}
> yarn app -flex simple-aa-1 -component ping 5
> {code}
> This is what is seen in the serviceam log at this point -
> {noformat}
> 2018-05-03 20:17:38,469 [IPC Server handler 0 on 38124] INFO  service.ClientAMService
- Flexing component ping to 5
> 2018-05-03 20:17:38,469 [Component  dispatcher] INFO  component.Component - [FLEX DOWN
COMPONENT ping]: scaling down from 6 to 5
> 2018-05-03 20:17:38,470 [Component  dispatcher] INFO  instance.ComponentInstance - [COMPINSTANCE
ping-4 : container_1525297086734_0013_01_000006]: Flexed down by user, destroying.
> 2018-05-03 20:17:38,473 [Component  dispatcher] INFO  component.Component - [COMPONENT
ping] Transitioned from FLEXING to STABLE on FLEX event.
> 2018-05-03 20:17:38,474 [pool-5-thread-8] INFO  registry.YarnRegistryViewForProviders
- [COMPINSTANCE ping-4 : container_1525297086734_0013_01_000006]: Deleting registry path /users/root/services/yarn-service/simple-aa-1/components/ctr-1525297086734-0013-01-000006
> 2018-05-03 20:17:38,476 [Component  dispatcher] ERROR component.Component - [COMPONENT
ping]: Invalid event CHECK_STABLE at STABLE
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: CHECK_STABLE
at STABLE
> 	at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:388)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> 	at org.apache.hadoop.yarn.service.component.Component.handle(Component.java:913)
> 	at org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:574)
> 	at org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:563)
> 	at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> 	at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> 	at java.lang.Thread.run(Thread.java:745)
> 2018-05-03 20:17:38,480 [Component  dispatcher] ERROR component.Component - [COMPONENT
ping]: Invalid event CHECK_STABLE at STABLE
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: CHECK_STABLE
at STABLE
> 	at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:388)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> 	at org.apache.hadoop.yarn.service.component.Component.handle(Component.java:913)
> 	at org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:574)
> 	at org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:563)
> 	at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> 	at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> 	at java.lang.Thread.run(Thread.java:745)
> 2018-05-03 20:17:38,578 [pool-5-thread-8] INFO  instance.ComponentInstance - [COMPINSTANCE
ping-4 : container_1525297086734_0013_01_000006]: Deleted component instance dir: hdfs://ctr-e138-1518143905142-280820-01-000003.example.site:8020/user/root/.yarn/services/simple-aa-1/components/1/ping/ping-4
> 2018-05-03 20:17:39,268 [AMRM Callback Handler Thread] WARN  service.ServiceScheduler
- Container container_1525297086734_0013_01_000006 Completed. No component instance exists.
exitStatus=-100. diagnostics=Container released by application 
> 2018-05-03 20:17:40,273 [AMRM Callback Handler Thread] INFO  service.ServiceScheduler
- 1 containers allocated. 
> 2018-05-03 20:17:40,273 [AMRM Callback Handler Thread] INFO  service.ServiceScheduler
- [COMPONENT ping]: remove 0 outstanding container requests for allocateId 0
> 2018-05-03 20:17:40,274 [Component  dispatcher] INFO  component.Component - [COMPONENT
ping]: container_1525297086734_0013_01_000007 allocated, num pending component instances reduced
to 0
> 2018-05-03 20:17:40,274 [Component  dispatcher] INFO  component.Component - [COMPONENT
ping]: Assigned container_1525297086734_0013_01_000007 to component instance ping-5 and launch
on host ctr-e138-1518143905142-280820-01-000008.example.site:25454 
> 2018-05-03 20:17:40,277 [pool-6-thread-6] INFO  provider.ProviderUtils - [COMPINSTANCE
ping-5 : container_1525297086734_0013_01_000007]: Creating dir on hdfs: hdfs://ctr-e138-1518143905142-280820-01-000003.example.site:8020/user/root/.yarn/services/simple-aa-1/components/1/ping/ping-5
> 2018-05-03 20:17:40,316 [pool-6-thread-6] INFO  containerlaunch.ContainerLaunchService
- launching container container_1525297086734_0013_01_000007
> 2018-05-03 20:17:40,318 [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl
#5] INFO  impl.NMClientAsyncImpl - Processing Event EventType: START_CONTAINER for Container
container_1525297086734_0013_01_000007
> 2018-05-03 20:17:40,338 [Component  dispatcher] ERROR component.Component - [COMPONENT
ping]: Invalid event CONTAINER_STARTED at STABLE
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: CONTAINER_STARTED
at STABLE
> 	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> 	at org.apache.hadoop.yarn.service.component.Component.handle(Component.java:913)
> 	at org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:574)
> 	at org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:563)
> 	at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> 	at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Status response shows that only 4 containers are running and the service is not in STABLE
state -
> {code:java}
> yarn app -status simple-aa-1
> {code}
> output -
> {code:java}
> {
>     "components": [
>         {
>             "configuration": {
>                 "env": {},
>                 "files": [],
>                 "properties": {}
>             },
>             "containers": [
>                 {
>                     "bare_host": "ctr-e138-1518143905142-280820-01-000007.example.site",
>                     "component_instance_name": "ping-1",
>                     "hostname": "ctr-e138-1518143905142-280820-01-000007.example.site",
>                     "id": "container_1525297086734_0013_01_000003",
>                     "ip": "x.x.x.x",
>                     "launch_time": 1525378141535,
>                     "state": "READY"
>                 },
>                 {
>                     "bare_host": "ctr-e138-1518143905142-280820-01-000006.example.site",
>                     "component_instance_name": "ping-0",
>                     "hostname": "ctr-e138-1518143905142-280820-01-000006.example.site",
>                     "id": "container_1525297086734_0013_01_000002",
>                     "ip": "x.x.x.x",
>                     "launch_time": 1525378141513,
>                     "state": "READY"
>                 },
>                 {
>                     "bare_host": "ctr-e138-1518143905142-280820-01-000005.example.site",
>                     "component_instance_name": "ping-3",
>                     "hostname": "ctr-e138-1518143905142-280820-01-000005.example.site",
>                     "id": "container_1525297086734_0013_01_000005",
>                     "ip": "x.x.x.x",
>                     "launch_time": 1525378303429,
>                     "state": "READY"
>                 },
>                 {
>                     "bare_host": "ctr-e138-1518143905142-280820-01-000004.example.site",
>                     "component_instance_name": "ping-2",
>                     "hostname": "ctr-e138-1518143905142-280820-01-000004.example.site",
>                     "id": "container_1525297086734_0013_01_000004",
>                     "ip": "x.x.x.x",
>                     "launch_time": 1525378303425,
>                     "state": "READY"
>                 }
>             ],
>             "dependencies": [],
>             "launch_command": "sleep 9000",
>             "name": "ping",
>             "number_of_containers": 5,
>             "placement_policy": {
>                 "constraints": [
>                     {
>                         "node_attributes": {},
>                         "node_partitions": [],
>                         "scope": "NODE",
>                         "target_tags": [
>                             "ping"
>                         ],
>                         "type": "ANTI_AFFINITY"
>                     }
>                 ]
>             },
>             "quicklinks": [],
>             "resource": {
>                 "additional": {},
>                 "cpus": 1,
>                 "memory": "256"
>             },
>             "run_privileged_container": false,
>             "state": "FLEXING"
>         }
>     ],
>     "configuration": {
>         "env": {},
>         "files": [],
>         "properties": {}
>     },
>     "id": "application_1525297086734_0013",
>     "kerberos_principal": {},
>     "lifetime": -1,
>     "name": "simple-aa-1",
>     "quicklinks": {},
>     "state": "STARTED",
>     "version": "1"
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message