hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mostafa Elhemali (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-729) MR uses LD_LIBRARY_PATH which doesn't mean anything in Windows
Date Mon, 24 Jun 2013 21:34:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13692409#comment-13692409
] 

Mostafa Elhemali commented on YARN-729:
---------------------------------------

Chuan: PATH and LD_LIBRARY_PATH can also be used for JNI libraries - it's not very well documented
so I can't find a link off a casual search, but you can test this behavior yourself. Typically
yes, we do use java.library.path, but in MR for some reason we rely on LD_LIBRARY_PATH not
java.library.path, so it doesn't work (see MRJobConfig.java, DEFAULT_MAPRED_ADMIN_USER_ENV).
I suspect you don't see this error because you just happen to have the native hadoop.dll in
one of the directories in your PATH already, but if you don't have that then you'll run into
this problem. Manually setting mapreduce.admin.user.env works around this problem in Windows,
but we should fix that so by default it runs properly on Windows.
                
> MR uses LD_LIBRARY_PATH which doesn't mean anything in Windows
> --------------------------------------------------------------
>
>                 Key: YARN-729
>                 URL: https://issues.apache.org/jira/browse/YARN-729
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>         Environment: Windows
>            Reporter: Mostafa Elhemali
>
> In order to set the path for loading native libraries, MR relies on the default value
of the mapreduce.admin.user.env configuration setting the LD_LIBRARY_PATH environment entry.
There are two problems with this setting in Windows:
> a) LD_LIBRARY_PATH doesn't mean anything in Windows.
> b) It sets it using $HADOOP_COMMON_HOME, instead of %HADOOP_COMMON_HOME%.
> The default value here should be platform-dependent (use the PATH variable in Windows
instead of LD_LIBRARY_PATH), or we should rely on another mechanism. The net effect is that
in Windows unless this configuration is over-ridden MR jobs fail with this error:
> {code}
> 2013-05-29 13:51:41,049 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running
child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
> 	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
> 	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:393)
> 	at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:928)
> 	at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:177)
> 	at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:164)
> 	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:98)
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:288)
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:431)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
> 	at org.apache.hadoop.mapred.YarnChild.configureLocalDirs(YarnChild.java:235)
> 	at org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:294)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message