hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (YARN-4502) Sometimes Two AM containers get launched
Date Wed, 13 Jan 2016 01:12:40 GMT

     [ https://issues.apache.org/jira/browse/YARN-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Vinod Kumar Vavilapalli reassigned YARN-4502:

    Assignee: Vinod Kumar Vavilapalli  (was: Wangda Tan)

bq. If add-application-attempt-event sent to scheduler before container-rescheduled-event
arrives, application attempt will be replaced so resource request will be restored to next
Seemed like a wild theory on first look, but it isn't and you are right! Because the current
flow is {{ContainerRescheduledTransition -> RM level event handler + queue -> Scheduler
event handler + queue}}, it is actually very likely for this to happen.

Once we remove this flow and let scheduler do the {{kill-container + recover-requests}} in
one shot, none of the routing-to-the wrong attempt will happen anymore.

Let me take a crack at this, assigning this to myself.

Side issue discovered: While the app-attempt finish is in the process of saving to state-store,
the scheduler can happily go around allocating more and more containers for the (finishing)

> Sometimes Two AM containers get launched
> ----------------------------------------
>                 Key: YARN-4502
>                 URL: https://issues.apache.org/jira/browse/YARN-4502
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Yesha Vora
>            Assignee: Vinod Kumar Vavilapalli
>            Priority: Critical
> Scenario : 
> * set yarn.resourcemanager.am.max-attempts = 2
> * start dshell application
> {code}
>  yarn  org.apache.hadoop.yarn.applications.distributedshell.Client -jar hadoop-yarn-applications-distributedshell-*.jar
-attempt_failures_validity_interval 60000 -shell_command "sleep 150" -num_containers 16
> {code}
> * Kill AM pid
> * Print container list for 2nd attempt
> {code}
> yarn container -list appattempt_1450825622869_0001_000002
> INFO impl.TimelineClientImpl: Timeline service address: http://xxx:port/ws/v1/timeline/
> INFO client.RMProxy: Connecting to ResourceManager at xxx/<port>
> Total number of containers :2
> Container-Id                 Start Time             Finish Time                   State
                   Host       Node Http Address                                LOG-URL
> container_e12_1450825622869_0001_02_000002 Tue Dec 22 23:07:35 +0000 2015           
       N/A                 RUNNING    xxx:25454       http://xxx:8042 http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_000002/hrt_qa
> container_e12_1450825622869_0001_02_000001 Tue Dec 22 23:07:34 +0000 2015           
       N/A                 RUNNING    xxx:25454       http://xxx:8042 http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_000001/hrt_qa
> {code}
> * look for new AM pid 
> Here, 2nd AM container was suppose to be started on  container_e12_1450825622869_0001_02_000001.
But AM was not launched on container_e12_1450825622869_0001_02_000001. It was in AQUIRED state.

> On other hand, container_e12_1450825622869_0001_02_000002 got the AM running. 
> Expected behavior: RM should not start 2 containers for starting AM

This message was sent by Atlassian JIRA

View raw message