hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chuan Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-729) MR uses LD_LIBRARY_PATH which doesn't mean anything in Windows
Date Thu, 30 May 2013 21:02:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670712#comment-13670712
] 

Chuan Liu commented on YARN-729:
--------------------------------

Hi [~mostafae], how did you run into this exception?

I agree mapreduce.admin.user.env default value should be OS dependent.
But I think this is not a fix to the exception you saw.

LD_LIBRARY_PATH and PATH are used for OS functions to dynamically load libraries, i.e. they
are for dlopen() and LoadLibrary() on Linux and Windows respectively.
On the other hand, the JNI libraries or DLLs load path is configured via the Java property
-Djava.library.path, which is used by Java API System.loadLibrary (not to be confused with
Windows LoadLibrary()).
I think the exception you saw was caused by the second API failure, i.e. native library is
not loaded, instead of a foreign library missing on PATH.
To confirm, you could check if you have following warnings in your log entries prior to the
exception.
{noformat}
Unable to load native-hadoop library for your platform...
{noformat}
                
> MR uses LD_LIBRARY_PATH which doesn't mean anything in Windows
> --------------------------------------------------------------
>
>                 Key: YARN-729
>                 URL: https://issues.apache.org/jira/browse/YARN-729
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>         Environment: Windows
>            Reporter: Mostafa Elhemali
>
> In order to set the path for loading native libraries, MR relies on the default value
of the mapreduce.admin.user.env configuration setting the LD_LIBRARY_PATH environment entry.
There are two problems with this setting in Windows:
> a) LD_LIBRARY_PATH doesn't mean anything in Windows.
> b) It sets it using $HADOOP_COMMON_HOME, instead of %HADOOP_COMMON_HOME%.
> The default value here should be platform-dependent (use the PATH variable in Windows
instead of LD_LIBRARY_PATH), or we should rely on another mechanism. The net effect is that
in Windows unless this configuration is over-ridden MR jobs fail with this error:
> {code}
> 2013-05-29 13:51:41,049 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running
child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
> 	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
> 	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:393)
> 	at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:928)
> 	at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:177)
> 	at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:164)
> 	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:98)
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:288)
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:431)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
> 	at org.apache.hadoop.mapred.YarnChild.configureLocalDirs(YarnChild.java:235)
> 	at org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:294)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message