hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1025) NodeManager does not propagate java.library.path to launched child containers on Windows
Date Sun, 04 Aug 2013 19:01:49 GMT

    [ https://issues.apache.org/jira/browse/YARN-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13728957#comment-13728957
] 

Chris Nauroth commented on YARN-1025:
-------------------------------------

As a workaround, you can set the PATH environment variable to point to the directory containing
hadoop.dll before launching NodeManager.  The environment variable will propagate to child
container processes, so this works.

I have a question on how this is expected to work on Linux.  I see code in the {{yarn}} shell
script and {{LinuxContainerExecutor}} that propagates java.library.path to children, but I
also see checks in {{YARNRunner#warnForJavaLibPath}} that warn if a caller is attempting to
control java.library.path and state that the caller should set LD_LIBRARY_PATH instead.  What
is the expected way for this to work on Linux?

If the expectation is for the caller to set LD_LIBRARY_PATH, then this issue is likely a Won't
Fix.  Setting PATH on Windows is roughly equivalent to setting LD_LIBRARY_PATH on Linux. 
I'm already using this technique successfully as a workaround.

If the expectation is that the shell scripts set up java.library.path, and NodeManager propagates
that to its children, then we do have a bug on Windows and need to make some code changes.
 The first part is changing {{yarn.cmd}} to set java.library.path.  That's a small easy change
that gets NodeManager to load hadoop.dll.  To cover the child processes, we'll need additional
code changes, probably in {{DefaultContainerExecutor}}.
                
> NodeManager does not propagate java.library.path to launched child containers on Windows
> ----------------------------------------------------------------------------------------
>
>                 Key: YARN-1025
>                 URL: https://issues.apache.org/jira/browse/YARN-1025
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 3.0.0, 2.1.1-beta
>            Reporter: Chris Nauroth
>
> Neither the NodeManager process itself nor the child container processes that it launches
have the correct setting for java.library.path on Windows.  This prevents the processes from
loading native code from hadoop.dll.  The native code is required for correct functioning
on Windows (not optional), so this ultimately can cause failures in MapReduce jobs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message