Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F2118D5FE for ; Thu, 14 Mar 2013 18:26:17 +0000 (UTC) Received: (qmail 2724 invoked by uid 500); 14 Mar 2013 18:26:16 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 2689 invoked by uid 500); 14 Mar 2013 18:26:16 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 2653 invoked by uid 99); 14 Mar 2013 18:26:16 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Mar 2013 18:26:16 +0000 Date: Thu, 14 Mar 2013 18:26:16 +0000 (UTC) From: "Zhijie Shen (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-367) Exception when yarn.nodemanager.local-dirs is not explicitly set MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602546#comment-13602546 ] Zhijie Shen commented on YARN-367: ---------------------------------- @Vinod, I've verified it. hadoop.tmp.dir was visible to YARN when it was put into core-site.xml, and the exception was gone. > Exception when yarn.nodemanager.local-dirs is not explicitly set > ---------------------------------------------------------------- > > Key: YARN-367 > URL: https://issues.apache.org/jira/browse/YARN-367 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager > Reporter: Zhijie Shen > Assignee: Zhijie Shen > > If yarn.nodemanager.local-dirs is not explicitly set, and if the default local-dirs are not the children of hadoop.tmp.dir, the exception will occur when the wordcount example is run. Bellow is log info. > ========== > 2013-01-30 22:16:04,229 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1359612879014_0001_01_000001 by user zshen > 2013-01-30 22:16:04,247 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1359612879014_0001 > 2013-01-30 22:16:04,250 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=zshen IP=127.0.0.1 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1359612879014_0001 CONTAINERID=container_1359612879014_0001_01_000001 > 2013-01-30 22:16:04,252 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1359612879014_0001 transitioned from NEW to INITING > 2013-01-30 22:16:04,252 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1359612879014_0001_01_000001 to application application_1359612879014_0001 > 2013-01-30 22:16:04,257 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1359612879014_0001 transitioned from INITING to RUNNING > 2013-01-30 22:16:04,262 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1359612879014_0001_01_000001 transitioned from NEW to LOCALIZING > 2013-01-30 22:16:04,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://localhost:9001/tmp/hadoop-yarn/staging/zshen/.staging/job_1359612879014_0001/appTokens transitioned from INIT to DOWNLOADING > 2013-01-30 22:16:04,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://localhost:9001/tmp/hadoop-yarn/staging/zshen/.staging/job_1359612879014_0001/job.jar transitioned from INIT to DOWNLOADING > 2013-01-30 22:16:04,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://localhost:9001/tmp/hadoop-yarn/staging/zshen/.staging/job_1359612879014_0001/job.splitmetainfo transitioned from INIT to DOWNLOADING > 2013-01-30 22:16:04,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://localhost:9001/tmp/hadoop-yarn/staging/zshen/.staging/job_1359612879014_0001/job.split transitioned from INIT to DOWNLOADING > 2013-01-30 22:16:04,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://localhost:9001/tmp/hadoop-yarn/staging/zshen/.staging/job_1359612879014_0001/job.xml transitioned from INIT to DOWNLOADING > 2013-01-30 22:16:04,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1359612879014_0001_01_000001 > 2013-01-30 22:16:04,401 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-zshen/nm-local-dir/nmPrivate/container_1359612879014_0001_01_000001.tokens. Credentials list: > 2013-01-30 22:16:04,423 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user zshen > 2013-01-30 22:16:04,569 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-zshen/nm-local-dir/nmPrivate/container_1359612879014_0001_01_000001.tokens to /tmp/hadoop-zshen/nm-local-dir/usercache/zshen/appcache/application_1359612879014_0001/container_1359612879014_0001_01_000001.tokens > 2013-01-30 22:16:04,570 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set to /tmp/hadoop-zshen/nm-local-dir/usercache/zshen/appcache/application_1359612879014_0001 = file:/tmp/hadoop-zshen/nm-local-dir/usercache/zshen/appcache/application_1359612879014_0001 > 2013-01-30 22:16:04,955 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out status for container: container_id {, app_attempt_id {, application_id {, id: 1, cluster_timestamp: 1359612879014, }, attemptId: 1, }, id: 1, }, state: C_RUNNING, diagnostics: "", exit_status: -1000, > 2013-01-30 22:16:05,117 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://localhost:9001/tmp/hadoop-yarn/staging/zshen/.staging/job_1359612879014_0001/appTokens transitioned from DOWNLOADING to LOCALIZED > 2013-01-30 22:16:05,312 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://localhost:9001/tmp/hadoop-yarn/staging/zshen/.staging/job_1359612879014_0001/job.jar transitioned from DOWNLOADING to LOCALIZED > 2013-01-30 22:16:05,465 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://localhost:9001/tmp/hadoop-yarn/staging/zshen/.staging/job_1359612879014_0001/job.splitmetainfo transitioned from DOWNLOADING to LOCALIZED > 2013-01-30 22:16:05,608 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://localhost:9001/tmp/hadoop-yarn/staging/zshen/.staging/job_1359612879014_0001/job.split transitioned from DOWNLOADING to LOCALIZED > 2013-01-30 22:16:05,751 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://localhost:9001/tmp/hadoop-yarn/staging/zshen/.staging/job_1359612879014_0001/job.xml transitioned from DOWNLOADING to LOCALIZED > 2013-01-30 22:16:05,752 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1359612879014_0001_01_000001 transitioned from LOCALIZING to LOCALIZED > 2013-01-30 22:16:05,866 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1359612879014_0001_01_000001 transitioned from LOCALIZED to RUNNING > 2013-01-30 22:16:05,866 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: ResourceCalculatorPlugin is unavailable on this system. org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is disabled. > 2013-01-30 22:16:05,910 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Failed to launch container. > java.io.FileNotFoundException: File /Users/zshen/Deployment/hadoop-3.0.0-SNAPSHOT/data/nm-local-dir/usercache/zshen/appcache/application_1359612879014_0001/container_1359612879014_0001_01_000001 does not exist > at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:498) > at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:996) > at org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150) > at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187) > at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:730) > at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:726) > at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2379) > at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:726) > at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createDir(DefaultContainerExecutor.java:330) > at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:135) > at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:242) > at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:680) > 2013-01-30 22:16:05,913 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1359612879014_0001_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE > 2013-01-30 22:16:05,914 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1359612879014_0001_01_000001 > 2013-01-30 22:16:05,934 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-zshen/nm-local-dir/usercache/zshen/appcache/application_1359612879014_0001/container_1359612879014_0001_01_000001 > 2013-01-30 22:16:05,934 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=zshen OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1359612879014_0001 CONTAINERID=container_1359612879014_0001_01_000001 > 2013-01-30 22:16:05,937 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1359612879014_0001_01_000001 transitioned from EXITED_WITH_FAILURE to DONE > 2013-01-30 22:16:05,937 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1359612879014_0001_01_000001 from application application_1359612879014_0001 > 2013-01-30 22:16:05,937 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: ResourceCalculatorPlugin is unavailable on this system. org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is disabled. > 2013-01-30 22:16:05,958 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out status for container: container_id {, app_attempt_id {, application_id {, id: 1, cluster_timestamp: 1359612879014, }, attemptId: 1, }, id: 1, }, state: C_COMPLETE, diagnostics: "", exit_status: -1, > 2013-01-30 22:16:05,959 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed container container_1359612879014_0001_01_000001 > 2013-01-30 22:16:06,965 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1359612879014_0001 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP > 2013-01-30 22:16:06,965 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-zshen/nm-local-dir/usercache/zshen/appcache/application_1359612879014_0001 > 2013-01-30 22:16:06,966 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1359612879014_0001 > 2013-01-30 22:16:06,970 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1359612879014_0001 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED > 2013-01-30 22:16:06,970 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1359612879014_0001, with delay of 10800 seconds > ========== > Below is the setting in hdfs-site.xml. > ========== > > hadoop.tmp.dir > /Users/zshen/Deployment/hadoop-3.0.0-SNAPSHOT/data > > ========== -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira