hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4072) User set java.library.path seems to overwrite default creating problems native lib loading
Date Thu, 29 Mar 2012 14:46:29 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241278#comment-13241278

Kihwal Lee commented on MAPREDUCE-4072:

I tested a bit.  

If we are to rely on LD_LIBRARY_PATH, there should be NO -Djava.library.path. Once this property
is set, the JVM only searches the paths in java.library.path for loading JNI libraries and
ignores LD_LIBRARY_PATH. I think this was the design in 0.23. It means that if users supply
-Djava.library.path in client map/red options, something can break.

Using LD_LIBRARY_PATH is better since the JVM relies on the system function to locate the
correct library, whereas specifying java.library.path causes the JVM's own resource loader
to be in charge, which can be troublesome if the path contains a library with the same name
but for different architecture.

1. The bug of unconditionally specifying -Djava.library.path for child jvm needs to be fixed.
2. Educate users that use of -Djava.library.path is no longer supported. They should specify
LD_LIBRARY_PATH in the child map/red jvm env config.  This is an incompatible change in 0.23.

If we want to support java.library.path, we have to reintroduce the java.library.path munging
code. This will make the system compatible, but has its own share of problems and not is against
the goal of the YARN architecture.

We should avoid setting -Djava.library.path when launching container and instead set LD_LIBRARY_PATH
> User set java.library.path seems to overwrite default creating problems native lib loading
> ------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-4072
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Anupam Seth
>            Assignee: Anupam Seth
> This was found by Peeyush Bishnoi.
> While running a distributed cache example with Hadoop-0.23,
> tasks are failing as follows:
> ------------------------------------------------------------------------------------------------------------
> Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException: at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at
> org.apache.hadoop.util.Shell.run(Shell.java:188) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
> java.util.concurrent.FutureTask.run(FutureTask.java:138) at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user
> is <user>
> ------------------------------------------------------------------------------------------------------------
> Same Pig script and command work successfully on 0.20
> See this in the stderr:
> Exception in thread "main" java.lang.ExceptionInInitializerError
>     at java.lang.Class.forName0(Native Method)
>     at java.lang.Class.forName(Class.java:247)
>     at
> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179)
>     at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149)
>     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238)
>     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264)
>     at org.apache.hadoop.security.Groups.(Groups.java:54)
>     at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178)
>     at
> org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
>     at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
>     at
> org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265)
>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75)
> Caused by: java.lang.RuntimeException: Bailing out since native library
> couldn't be loaded
>     at
> org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48)
>     ... 12 more
> Pig command:
> $ pig -Dmapred.job.queue.name=<queue> -Dmapred.cache.archives=<archives>
> -Dip2geo.preLoadLibraries=<some other libs>" -Djava.io.tmpdir=/grid/0/tmp -Dmapred.create.symlink=yes
-Dmapred.job.map.memory.mb=3072 piggeoscript.pig

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message