hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3259) ContainerLocalizer should get the proper java.library.path from LinuxContainerExecutor
Date Wed, 26 Oct 2011 21:41:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136441#comment-13136441

Kihwal Lee commented on MAPREDUCE-3259:

bq. Kihwal, won't it suffice to have the right configs (and defaults) for admin.env?
For executing the task script, that's true.  For LTC directly launching JVM to run localizer,
either LD_LIBRARY_PATH or -Djava.library.path is need to be set properly. It will appear to
work until something requiring the native lib is enabled. When I enabled the JNI-based user
groups mapping feature, the localizer wouldn't run.

Back to running task script. The task jvm may have to load libhadoop.so because of a certain
cluster-level config. I think we should treat it as an app-level dependency, rather than a
run-time dependency of specific client code. Sometimes even admins won't know whether the
library will be required by enabling a feature. I thought it will be simpler to add java.library.path
to take care of the dependency at the app-level.

> ContainerLocalizer should get the proper java.library.path from LinuxContainerExecutor
> --------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-3259
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3259
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2, nodemanager
>    Affects Versions: 0.23.0, 0.24.0
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Blocker
>             Fix For: 0.23.0, 0.24.0
>         Attachments: mapreduce-3259.patch, mapreduce-3259.patch.txt
> As seen in MAPREDUCE-2915, java.library.path is not being passed when the LCE spawns
a JVM for ContainerLocalizer. 
> However, unlike branch-0.20-security, the task runtime in 0.23 is unaffected by this.
This is because tasks' run-time environment is specified in the launch script by client. Setting
LD_LIBRARY_PATH is the primary way of specifying the locations of required native library
in this case. The config property, mapreduce.admin.user.env is always set in the job environment
and the default value is to add the path to the hadoop native library to LD_LABRARY_PATH.
> For JVM's being launched by the hadoop system scripts, java.library.path is set.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message