hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2057) NPE in RM handling node update while app submission in progress
Date Wed, 14 May 2014 13:59:15 GMT

    [ https://issues.apache.org/jira/browse/YARN-2057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997557#comment-13997557
] 

Steve Loughran commented on YARN-2057:
--------------------------------------

Stack trace. This is a transient failure -and didn't reoccur immediately after- so the log
is all there is, I'm afraid

Can I note that failing to handle status updates by triggering RM exit is a bit brittle. It
exposes the RM to failover if a single NM starts sending in bad data.

{code}
2014-05-14 14:48:32,248 [JUnit] DEBUG launch.AbstractLauncher (AbstractLauncher.java:completeContainerLaunch(162))
- Completed setting up container command $JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true
-Djava.awt.headless=true -Xmx256M -ea -esa org.apache.slider.server.appmaster.SliderAppMaster
create test_destroy_masterless_am --debug -cluster-uri file:/Users/stevel/.slider/cluster/test_destroy_masterless_am
--rm 192.168.1.86:54470 --fs file:/// -D slider.registry.path=/registry -D slider.zookeeper.quorum=localhost:1
1><LOG_DIR>/slider-out.txt 2><LOG_DIR>/slider-err.txt 
2014-05-14 14:48:32,249 [JUnit] INFO  launch.AppMasterLauncher (AppMasterLauncher.java:submitApplication(207))
- Submitting application to Resource Manager
2014-05-14 14:48:32,281 [IPC Server handler 2 on 54471] INFO  resourcemanager.ClientRMService
(ClientRMService.java:submitApplication(537)) - Application with id 1 submitted by user stevel
2014-05-14 14:48:32,281 [AsyncDispatcher event handler] INFO  rmapp.RMAppImpl (RMAppImpl.java:transition(863))
- Storing application with id application_1400075308869_0001
2014-05-14 14:48:32,282 [IPC Server handler 2 on 54471] INFO  resourcemanager.RMAuditLogger
(RMAuditLogger.java:logSuccess(142)) - USER=stevel	IP=192.168.1.86	OPERATION=Submit Application
Request	TARGET=ClientRMService	RESULT=SUCCESS	APPID=application_1400075308869_0001
2014-05-14 14:48:32,283 [AsyncDispatcher event handler] INFO  rmapp.RMAppImpl (RMAppImpl.java:handle(639))
- application_1400075308869_0001 State change from NEW to NEW_SAVING
2014-05-14 14:48:32,287 [AsyncDispatcher event handler] INFO  recovery.RMStateStore (RMStateStore.java:handleStoreEvent(620))
- Storing info for app: application_1400075308869_0001
2014-05-14 14:48:32,295 [AsyncDispatcher event handler] INFO  rmapp.RMAppImpl (RMAppImpl.java:handle(639))
- application_1400075308869_0001 State change from NEW_SAVING to SUBMITTED
2014-05-14 14:48:32,296 [ResourceManager Event Processor] INFO  fifo.FifoScheduler (FifoScheduler.java:addApplication(369))
- Accepted application application_1400075308869_0001 from user: stevel, currently num of
applications: 1
2014-05-14 14:48:32,297 [ResourceManager Event Processor] FATAL resourcemanager.ResourceManager
(ResourceManager.java:run(600)) - Error in handling event type NODE_UPDATE to the scheduler
java.lang.NullPointerException
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.assignContainers(FifoScheduler.java:462)
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.nodeUpdate(FifoScheduler.java:714)
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.handle(FifoScheduler.java:743)
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.handle(FifoScheduler.java:104)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:591)
	at java.lang.Thread.run(Thread.java:745)
2014-05-14 14:48:32,298 [ResourceManager Event Processor] INFO  resourcemanager.ResourceManager
(ResourceManager.java:run(604)) - Exiting, bbye..
2014-05-14 14:48:32,298 [AsyncDispatcher event handler] INFO  rmapp.RMAppImpl (RMAppImpl.java:handle(639))
- application_1400075308869_0001 State change from SUBMITTED to ACCEPTED
2014-05-14 14:48:32,299 [AsyncDispatcher event handler] INFO  resourcemanager.ApplicationMasterService
(ApplicationMasterService.java:registerAppAttempt(608)) - Registering app attempt : appattempt_1400075308869_0001_000001

{code}

> NPE in RM handling node update while app submission in progress
> ---------------------------------------------------------------
>
>                 Key: YARN-2057
>                 URL: https://issues.apache.org/jira/browse/YARN-2057
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.4.0
>         Environment: OS/X, hadoop 2.4.0 mini yarn cluster, slider unit test TestDestroyMasterlessAM
>            Reporter: Steve Loughran
>
> One of our test runs finished prematurely with an NPE in the RM, followed by the RM thread
calling system.exit(). It looks like an NM update came in while the app was still being set
up, causing confusion.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message