hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3259) ContainerLocalizer should get the proper java.library.path from LinuxContainerExecutor
Date Thu, 27 Oct 2011 00:35:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136642#comment-13136642

Kihwal Lee commented on MAPREDUCE-3259:

Sorry, my explanation is rather wrong and confusing. I shouldn't have said app-level. That's
totally wrong. 

Apps are fine as you said. It's about yarn system level env. When a JVM is directly launched
by the LTC, neither of LD_LIBRARY_PATH or java.library.path is set. This is the case for ContainerLocalizer.
It will fail to load libhadoop.so if it needs it. If the library is installed in one of the
directories ld looks at, it will work without the setting. Or the hadoop lib path may be added
to /etc/ld.so.conf or /etc/ld.so.conf.d.  Is it still confusing?  Or am I totally wrong about

I can leave out the second part in the patch for the client since a proper env can be placed
in the container script using the facility you mentioned. Only thing I don't like is the default,
which assumes the old dir layout.  I could try changing the default so that it has a better
chance of working right out of the box, but that's not critical.

> ContainerLocalizer should get the proper java.library.path from LinuxContainerExecutor
> --------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-3259
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3259
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2, nodemanager
>    Affects Versions: 0.23.0, 0.24.0
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Blocker
>             Fix For: 0.23.0, 0.24.0
>         Attachments: mapreduce-3259.patch, mapreduce-3259.patch.txt
> As seen in MAPREDUCE-2915, java.library.path is not being passed when the LCE spawns
a JVM for ContainerLocalizer. 
> However, unlike branch-0.20-security, the task runtime in 0.23 is unaffected by this.
This is because tasks' run-time environment is specified in the launch script by client. Setting
LD_LIBRARY_PATH is the primary way of specifying the locations of required native library
in this case. The config property, mapreduce.admin.user.env is always set in the job environment
and the default value is to add the path to the hadoop native library to LD_LABRARY_PATH.
> For JVM's being launched by the hadoop system scripts, java.library.path is set.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message