hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Drake민영근 <drake....@nexr.com>
Subject Re: ResourceLocalizationService: Localizer failed when running pi example
Date Mon, 20 Apr 2015 01:21:03 GMT
Hi,

guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
provide that part of yarn-site.xml?

Thanks.

Drake 민영근 Ph.D
kt NexR

On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fotero@gmail.com> wrote:

> yeah... there's not much there:
>
> -bash-4.1$ cd nm-local-dir/
> -bash-4.1$ ll *
> filecache:
> total 0
>
> nmPrivate:
> total 0
>
> usercache:
> total 0
>
> I'm using Open JDK, would that be a problem?
>
> More log:
>
> STARTUP_MSG:   java = 1.7.0_75
> ************************************************************/
> 2015-04-19 14:38:58,168 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
> signal handlers for [TERM, HUP, INT]
> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2015-04-19 14:38:59,018 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
> 2015-04-19 14:38:59,020 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
> 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
> 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
> 2015-04-19 14:38:59,022 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> 2015-04-19 14:38:59,023 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
> 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
> class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
> 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
> org.apache.hadoop.yarn.server.nodemanager.NodeManager
> 2015-04-19 14:38:59,109 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2015-04-19 14:38:59,197 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2015-04-19 14:38:59,197 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
> system started
> 2015-04-19 14:38:59,217 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
> 2015-04-19 14:38:59,217 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> per directory file limit = 8192
> 2015-04-19 14:38:59,227 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
> 2015-04-19 14:38:59,248 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
> 'httpshuffle'. Because these are not the same tools trying to send
> ServiceData and read Service Meta Data may have issues unless the refer to
> the name in the config.
> 2015-04-19 14:38:59,248 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  Using ResourceCalculatorPlugin :
> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  Using ResourceCalculatorProcessTree : null
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Physical memory check enabled: true
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Virtual memory check enabled: true
> 2015-04-19 14:38:59,284 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> NodeManager configured with 14 G physical memory allocated to containers,
> which is more than 80% of the total physical memory available (14.7 G).
> Thrashing might happen.
> 2015-04-19 14:38:59,287 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
> Initialized nodemanager for null: physical-memory=14336
> virtual-memory=30106 virtual-cores=8
> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager: Using
> callQueue class java.util.concurrent.LinkedBlockingQueue
> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 38230
> 2015-04-19 14:38:59,359 INFO
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
> server
> 2015-04-19 14:38:59,359 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Blocking new container-requests as container manager rpc server is still
> starting.
> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 38230: starting
> 2015-04-19 14:38:59,366 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
> Updating node address : ip-10-100-70-199.ec2.internal:38230
> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager: Using
> callQueue class java.util.concurrent.LinkedBlockingQueue
> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 8040
> 2015-04-19 14:38:59,376 INFO
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
> protocol
> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
> server
> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 8040: starting
> 2015-04-19 14:38:59,380 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer started on port 8040
> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
> IndexCache created with max memory = 10485760
> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
> httpshuffle listening on port 13562
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> ContainerManager bound to 0.0.0.0/0.0.0.0:0
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
> NMWebApp at 0.0.0.0:8042
> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
> request log for http.requests.nodemanager is not defined
> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
> global filter 'safety'
> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context node
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
> path spec: /node/*
> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
> path spec: /ws/*
> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
> bound to port 8042
> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
> app /node started at 8042
> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
> Registered webapp guice modules
> 2015-04-19 14:39:00,126 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
> out 0 NM container statuses: []
> 2015-04-19 14:39:00,131 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
> Registering with RM using containers :[]
> 2015-04-19 14:39:00,176 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
> Rolling master-key for container-tokens, got key with id -543066608
> 2015-04-19 14:39:00,178 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
> Rolling master-key for container-tokens, got key with id -1243797706
> 2015-04-19 14:39:00,179 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
> resource of <memory:14336, vCores:8>
> 2015-04-19 14:39:00,179 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
> ContainerManager to unblock new container-requests
> 2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
> Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
> 2015-04-19 19:22:17,807 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Start request for container_1429450734039_0010_01_000001 by user nobody
> 2015-04-19 19:22:17,828 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Creating a new application reference for app application_1429450734039_0010
> 2015-04-19 19:22:17,834 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
> IP=10.100.66.251 OPERATION=Start Container Request
> TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_1429450734039_0010
> CONTAINERID=container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,835 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Application application_1429450734039_0010 transitioned from NEW to INITING
> 2015-04-19 19:22:17,835 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Adding container_1429450734039_0010_01_000001 to application
> application_1429450734039_0010
> 2015-04-19 19:22:17,839 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Application application_1429450734039_0010 transitioned from INITING to
> RUNNING
> 2015-04-19 19:22:17,843 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from NEW to
> LOCALIZING
> 2015-04-19 19:22:17,843 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
> event CONTAINER_INIT for appId application_1429450734039_0010
> 2015-04-19 19:22:17,876 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Created localizer for container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,880 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
> 2015-04-19 19:22:17,882 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from
> LOCALIZING to LOCALIZATION_FAILED
> 2015-04-19 19:22:17,886 WARN
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
> failed with state: LOCALIZATION_FAILED
> APPID=application_1429450734039_0010
> CONTAINERID=container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,889 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from
> LOCALIZATION_FAILED to DONE
> 2015-04-19 19:22:17,889 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Removing container_1429450734039_0010_01_000001 from application
> application_1429450734039_0010
>
> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
> brahmareddy.battula@hotmail.com> wrote:
>
>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>> or mapred.local.dir)..
>>
>> can you able provide full logs..?
>>
>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>> details..
>>
>> ------------------------------
>> From: wget.null@gmail.com
>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>> pi example
>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>> To: user@hadoop.apache.org
>>
>>
>> As you said, that looks like a config issue. I would spot on the NM's
>> local scratch dir (yarn.nodemanager.local-dirs).
>>
>> But without a complete stack trace, its a blind call.
>>
>> BR,
>>  AL
>>
>> --
>> mapredit.blogspot.com
>>
>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fotero@gmail.com> wrote:
>>
>> Hey All,
>>     It's me again with another noob question: I deployed a cluster (HA
>> mode) everything looked good but when I tried to run the pi example:
>>
>>  bin/hadoop jar
>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>
>> the same error occurs if I try to generate data with teragen 100000000
>> /test/data
>>
>>
>> 2015-04-18 15:49:04,090 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer failed
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>
>>
>> I'm guessing it's a configuration issue but I don't know what am I
>> missing :S
>>
>>
>>
>

Mime
View raw message