hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kumar ranganathan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3120) YarnException on windows + org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to setup local dirnm-local-dir, which was marked as good.
Date Tue, 02 Feb 2016 13:35:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15128253#comment-15128253
] 

kumar ranganathan commented on YARN-3120:
-----------------------------------------

Hi,
I have facing the same problem in Hadoop-2.7.2 in Windows. I have found the logs in nodemanager
as below and it should be the actual root cause

{panel}
WARN localizer.ResourceLocalizationService: Failed to setup local dir /tmp/hadoop-SYSTEM/nm-local-dir,
which was marked as good.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Permissions incorrectly set for dir
/tmp/hadoop-SYSTEM/nm-local-dir/usercache, should be rwxr-xr-x, actual value = rwxrwxr-x
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.checkLocalDir(ResourceLocalizationService.java:1443)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.java:1410)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.access$800(ResourceLocalizationService.java:140)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1110)
16/02/02 18:50:45 INFO localizer.ResourceLocalizationService: Localizer failed
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to setup local dir /tmp/hadoop-SYSTEM/nm-local-dir,
which was marked as good.
{panel}

I found another jira task related to this issue
[https://issues.apache.org/jira/browse/YARN-3962]

Even though i applied patch, the problem isn't resolved. So i have changed the code manually
by commenting the condition check as below,

{code:title=ResourceLocalizationService.java|borderStyle=solid}
private boolean checkLocalDir(String localDir) {

    Map<Path, FsPermission> pathPermissionMap = getLocalDirsPathPermissionsMap(localDir);

    for (Map.Entry<Path, FsPermission> entry : pathPermissionMap.entrySet()) {
      FileStatus status;
      try {
        status = lfs.getFileStatus(entry.getKey());
      } catch (Exception e) {
        String msg =
            "Could not carry out resource dir checks for " + localDir
                + ", which was marked as good";
        LOG.warn(msg, e);
        throw new YarnRuntimeException(msg, e);
      }

//      if (!status.getPermission().equals(entry.getValue())) {
//        String msg =
//            "Permissions incorrectly set for dir " + entry.getKey()
//                + ", should be " + entry.getValue() + ", actual value = "
//                + status.getPermission();
//        LOG.warn(msg);
//        throw new YarnRuntimeException(msg);
//      }
    }
    return true;
  }
{code}

Now it works fine without any issue. It seems we can ignore this condition for virtual account
in windows.

> YarnException on windows + org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed
to setup local dirnm-local-dir, which was marked as good.
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-3120
>                 URL: https://issues.apache.org/jira/browse/YARN-3120
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>         Environment: Windows 8 , Hadoop 2.6.0
>            Reporter: vaidhyanathan
>
> Hi,
> I tried to follow the instructiosn in http://wiki.apache.org/hadoop/Hadoop2OnWindows
and have setup hadoop-2.6.0.jar in my windows system.
> I was able to start everything properly but when i try to run the job wordcount as given
in the above URL , the job fails with the below exception .
> 15/01/30 12:56:09 INFO localizer.ResourceLocalizationService: Localizer failed
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to setup local di
> r /tmp/hadoop-haremangala/nm-local-dir, which was marked as good.
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
> ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.
> java:1372)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
> ResourceLocalizationService.access$900(ResourceLocalizationService.java:137)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
> ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java
> :1085)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message