hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fernando O." <fot...@gmail.com>
Subject Re: ResourceLocalizationService: Localizer failed when running pi example
Date Mon, 20 Apr 2015 11:52:25 GMT
I didn't specify it so it's using the default value (in /tmp)

On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <drake.min@nexr.com> wrote:

> Hi,
>
> guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
> provide that part of yarn-site.xml?
>
> Thanks.
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fotero@gmail.com> wrote:
>
>> yeah... there's not much there:
>>
>> -bash-4.1$ cd nm-local-dir/
>> -bash-4.1$ ll *
>> filecache:
>> total 0
>>
>> nmPrivate:
>> total 0
>>
>> usercache:
>> total 0
>>
>> I'm using Open JDK, would that be a problem?
>>
>> More log:
>>
>> STARTUP_MSG:   java = 1.7.0_75
>> ************************************************************/
>> 2015-04-19 14:38:58,168 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
>> signal handlers for [TERM, HUP, INT]
>> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
>> Unable to load native-hadoop library for your platform... using
>> builtin-java classes where applicable
>> 2015-04-19 14:38:59,018 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
>> 2015-04-19 14:38:59,020 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
>> 2015-04-19 14:38:59,021 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>> 2015-04-19 14:38:59,021 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>> 2015-04-19 14:38:59,022 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>> 2015-04-19 14:38:59,023 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
>> 2015-04-19 14:38:59,054 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
>> class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>> 2015-04-19 14:38:59,054 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>> 2015-04-19 14:38:59,109 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2015-04-19 14:38:59,197 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2015-04-19 14:38:59,197 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>> system started
>> 2015-04-19 14:38:59,217 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
>> 2015-04-19 14:38:59,217 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> per directory file limit = 8192
>> 2015-04-19 14:38:59,227 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>> 2015-04-19 14:38:59,248 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
>> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
>> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
>> 'httpshuffle'. Because these are not the same tools trying to send
>> ServiceData and read Service Meta Data may have issues unless the refer to
>> the name in the config.
>> 2015-04-19 14:38:59,248 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
>> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>  Using ResourceCalculatorPlugin :
>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>  Using ResourceCalculatorProcessTree : null
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> Physical memory check enabled: true
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> Virtual memory check enabled: true
>> 2015-04-19 14:38:59,284 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> NodeManager configured with 14 G physical memory allocated to containers,
>> which is more than 80% of the total physical memory available (14.7 G).
>> Thrashing might happen.
>> 2015-04-19 14:38:59,287 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>> Initialized nodemanager for null: physical-memory=14336
>> virtual-memory=30106 virtual-cores=8
>> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager:
>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting
>> Socket Reader #1 for port 38230
>> 2015-04-19 14:38:59,359 INFO
>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
>> server
>> 2015-04-19 14:38:59,359 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Blocking new container-requests as container manager rpc server is still
>> starting.
>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 38230: starting
>> 2015-04-19 14:38:59,366 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>> Updating node address : ip-10-100-70-199.ec2.internal:38230
>> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager:
>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting
>> Socket Reader #1 for port 8040
>> 2015-04-19 14:38:59,376 INFO
>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>> protocol
>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
>> server
>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 8040: starting
>> 2015-04-19 14:38:59,380 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer started on port 8040
>> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
>> IndexCache created with max memory = 10485760
>> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
>> httpshuffle listening on port 13562
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> ContainerManager bound to 0.0.0.0/0.0.0.0:0
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
>> NMWebApp at 0.0.0.0:8042
>> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
>> request log for http.requests.nodemanager is not defined
>> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
>> global filter 'safety'
>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context node
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context logs
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context static
>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>> path spec: /node/*
>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>> path spec: /ws/*
>> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
>> bound to port 8042
>> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
>> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
>> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
>> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
>> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
>> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>> app /node started at 8042
>> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
>> Registered webapp guice modules
>> 2015-04-19 14:39:00,126 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
>> out 0 NM container statuses: []
>> 2015-04-19 14:39:00,131 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>> Registering with RM using containers :[]
>> 2015-04-19 14:39:00,176 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>> Rolling master-key for container-tokens, got key with id -543066608
>> 2015-04-19 14:39:00,178 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
>> Rolling master-key for container-tokens, got key with id -1243797706
>> 2015-04-19 14:39:00,179 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
>> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
>> resource of <memory:14336, vCores:8>
>> 2015-04-19 14:39:00,179 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
>> ContainerManager to unblock new container-requests
>> 2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
>> Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
>> 2015-04-19 19:22:17,807 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Start request for container_1429450734039_0010_01_000001 by user nobody
>> 2015-04-19 19:22:17,828 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Creating a new application reference for app application_1429450734039_0010
>> 2015-04-19 19:22:17,834 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
>> IP=10.100.66.251 OPERATION=Start Container Request
>> TARGET=ContainerManageImpl RESULT=SUCCESS
>> APPID=application_1429450734039_0010
>> CONTAINERID=container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,835 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Application application_1429450734039_0010 transitioned from NEW to INITING
>> 2015-04-19 19:22:17,835 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Adding container_1429450734039_0010_01_000001 to application
>> application_1429450734039_0010
>> 2015-04-19 19:22:17,839 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Application application_1429450734039_0010 transitioned from INITING to
>> RUNNING
>> 2015-04-19 19:22:17,843 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from NEW to
>> LOCALIZING
>> 2015-04-19 19:22:17,843 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
>> event CONTAINER_INIT for appId application_1429450734039_0010
>> 2015-04-19 19:22:17,876 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Created localizer for container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,880 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer failed
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>> 2015-04-19 19:22:17,882 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from
>> LOCALIZING to LOCALIZATION_FAILED
>> 2015-04-19 19:22:17,886 WARN
>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
>> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
>> failed with state: LOCALIZATION_FAILED
>> APPID=application_1429450734039_0010
>> CONTAINERID=container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,889 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from
>> LOCALIZATION_FAILED to DONE
>> 2015-04-19 19:22:17,889 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Removing container_1429450734039_0010_01_000001 from application
>> application_1429450734039_0010
>>
>> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
>> brahmareddy.battula@hotmail.com> wrote:
>>
>>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>>> or mapred.local.dir)..
>>>
>>> can you able provide full logs..?
>>>
>>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>>> details..
>>>
>>> ------------------------------
>>> From: wget.null@gmail.com
>>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>>> pi example
>>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>>> To: user@hadoop.apache.org
>>>
>>>
>>> As you said, that looks like a config issue. I would spot on the NM's
>>> local scratch dir (yarn.nodemanager.local-dirs).
>>>
>>> But without a complete stack trace, its a blind call.
>>>
>>> BR,
>>>  AL
>>>
>>> --
>>> mapredit.blogspot.com
>>>
>>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fotero@gmail.com> wrote:
>>>
>>> Hey All,
>>>     It's me again with another noob question: I deployed a cluster (HA
>>> mode) everything looked good but when I tried to run the pi example:
>>>
>>>  bin/hadoop jar
>>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>>
>>> the same error occurs if I try to generate data with teragen 100000000
>>> /test/data
>>>
>>>
>>> 2015-04-18 15:49:04,090 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Localizer failed
>>> java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>
>>>
>>> I'm guessing it's a configuration issue but I don't know what am I
>>> missing :S
>>>
>>>
>>>
>>
>

Mime
View raw message