bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julien Eid <julien....@cloudera.com>
Subject Re: Nodemanager not Starting
Date Fri, 25 Jul 2014 18:38:09 GMT
YarnException: Failed to check for existence of remoteLogDir
[/var/log/hadoop-yarn/apps]

I'm not very familiar with Yarn, but are you sure that directory exists?
Also https://issues.apache.org/jira/browse/YARN-24 states that the namenode
might be unreachable

There was also this link
https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/UOYbUcK_6q4
which will probably be more useful.

Sorry if this doesn't help.


On Fri, Jul 25, 2014 at 11:30 AM, David Fryer <dfryer1193@gmail.com> wrote:

> Hi Bigtop,
>
> I have used puppet to deploy Hadoop onto a master and three slaves,
> however on the slaves, nodemanager will not start.
>
> The errors found in /var/log/hadoop-yarn/yarn-yarn-nodemanager-slave1.log
> are here:
>
> 2014-07-26 01:24:22,370 ERROR
>> org.apache.hadoop.yarn.service.CompositeService: Error starting services
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>> org.apache.hadoop.yarn.YarnException: Failed to check for existence of
>> remoteLogDir [/var/log/hadoop-yarn/apps]
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:179)
>>  at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.start(LogAggregationService.java:132)
>> at
>> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
>>  at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.start(ContainerManagerImpl.java:248)
>> at
>> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
>>  at
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:199)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:322)
>>  at
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:359)
>> 2014-07-26 01:24:22,372 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService:
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService
>> waiting for pending aggregation during exit
>> 2014-07-26 01:24:22,372 INFO
>> org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is
>> stopped.
>> 2014-07-26 01:24:22,372 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>> is interrupted. Exiting.
>> 2014-07-26 01:24:22,372 INFO
>> org.apache.hadoop.yarn.service.AbstractService: Service:containers-monitor
>> is stopped.
>> 2014-07-26 01:24:22,379 INFO
>> org.apache.hadoop.yarn.service.AbstractService: Service:httpshuffle is
>> stopped.
>> 2014-07-26 01:24:22,379 INFO
>> org.apache.hadoop.yarn.service.AbstractService:
>> Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>> is stopped.
>> 2014-07-26 01:24:22,379 INFO
>> org.apache.hadoop.yarn.service.AbstractService: Service:containers-launcher
>> is stopped.
>> 2014-07-26 01:24:22,379 INFO org.apache.hadoop.ipc.Server: Stopping
>> server on 8040
>> 2014-07-26 01:24:22,380 INFO org.apache.hadoop.ipc.Server: Stopping IPC
>> Server listener on 8040
>> 2014-07-26 01:24:22,380 INFO org.apache.hadoop.ipc.Server: Stopping IPC
>> Server Responder
>> 2014-07-26 01:24:22,380 INFO
>> org.apache.hadoop.yarn.service.AbstractService:
>> Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>> is stopped.
>> 2014-07-26 01:24:22,380 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Public cache exiting
>> 2014-07-26 01:24:22,380 INFO
>> org.apache.hadoop.yarn.service.AbstractService:
>> Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>> is stopped.
>> 2014-07-26 01:24:22,380 ERROR
>> org.apache.hadoop.yarn.service.CompositeService: Error starting services
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>> org.apache.hadoop.yarn.YarnException: Failed to Start
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>> at
>> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
>>  at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.start(ContainerManagerImpl.java:248)
>> at
>> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
>>  at
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:199)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:322)
>>  at
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:359)
>> Caused by: org.apache.hadoop.yarn.YarnException: Failed to check for
>> existence of remoteLogDir [/var/log/hadoop-yarn/apps]
>>  at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:179)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.start(LogAggregationService.java:132)
>>  at
>> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
>> ... 5 more
>> 2014-07-26 01:24:22,380 INFO org.apache.hadoop.ipc.Server: Stopping
>> server on 42208
>> 2014-07-26 01:24:22,380 INFO org.apache.hadoop.ipc.Server: Stopping IPC
>> Server listener on 42208
>> 2014-07-26 01:24:22,381 INFO org.apache.hadoop.ipc.Server: Stopping IPC
>> Server Responder
>> 2014-07-26 01:24:22,380 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService:
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService
>> waiting for pending aggregation during exit
>> 2014-07-26 01:24:22,381 INFO org.apache.hadoop.ipc.Server: Stopping
>> server on 8040
>> 2014-07-26 01:24:22,381 INFO
>> org.apache.hadoop.yarn.service.AbstractService:
>> Service:org.apache.hadoop.yarn.server.nodemanager.NodeResourceMonitorImpl
>> is stopped.
>> 2014-07-26 01:24:22,381 INFO
>> org.apache.hadoop.yarn.service.AbstractService:
>> Service:org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService
>> is stopped.
>> 2014-07-26 01:24:22,381 INFO
>> org.apache.hadoop.yarn.service.AbstractService:
>> Service:org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService
>> is stopped.
>> 2014-07-26 01:24:22,381 INFO
>> org.apache.hadoop.yarn.service.AbstractService:
>> Service:org.apache.hadoop.yarn.server.nodemanager.DeletionService is
>> stopped.
>> 2014-07-26 01:24:22,381 FATAL
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
>> NodeManager
>> org.apache.hadoop.yarn.YarnException: Failed to Start
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>> at
>> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
>>  at
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:199)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:322)
>>  at
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:359)
>> Caused by: org.apache.hadoop.yarn.YarnException: Failed to Start
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>>  at
>> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.start(ContainerManagerImpl.java:248)
>>  at
>> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
>> ... 3 more
>> Caused by: org.apache.hadoop.yarn.YarnException: Failed to check for
>> existence of remoteLogDir [/var/log/hadoop-yarn/apps]
>>  at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:179)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.start(LogAggregationService.java:132)
>>  at
>> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
>> ... 5 more
>> 2014-07-26 01:24:22,382 INFO org.apache.hadoop.ipc.Server: Stopping
>> server on 42208
>> 2014-07-26 01:24:22,382 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService:
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService
>> waiting for pending aggregation during exit
>> 2014-07-26 01:24:22,382 INFO org.apache.hadoop.ipc.Server: Stopping
>> server on 8040
>> 2014-07-26 01:24:22,382 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager
>> metrics system...
>> 2014-07-26 01:24:22,383 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>> system stopped.
>> 2014-07-26 01:24:22,383 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>> system shutdown complete.
>> 2014-07-26 01:24:22,385 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down NodeManager at slave1/127.0.0.1
>> ************************************************************/
>
>
> What can I do to fix this?
>
> -David Fryer
>
>

Mime
View raw message