hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anupam Seth (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4072) User set java.library.path seems to overwrite default creating problems native lib loading
Date Mon, 02 Apr 2012 21:49:23 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244693#comment-13244693
] 

Anupam Seth commented on MAPREDUCE-4072:
----------------------------------------

One reason we might want to go for a doc change instead of stripping out and copying the java.library.path
set by the user to LD_LIBRARY_PATH is because it is cleaner to set it right in the first place
and we likely don't want the overhead of supporting such in the long run.

Quoting Kihwal to help clarify:
"If only LD_LIBRARY_PATH is specified, the JVM (at least's Oracle's) resorts to the dynamic
linker to search for the library. If java.library.path is set, the JVM will only look at the
paths in java.library.path.  Since the search capability of the linker is superior, we prefer
using the former method whenever possible.  The JVM resource loading stops searching when
it locates one match, whether the match is for the correct architecture or not. This can be
troublesome when supporting multiple architectures, which is a goal of yarn.

LD_LIBRARY_PATH is also needed if there is a chain of dependencies on native libraries from
a JNI library and those are placed in a non-standard directory(i.e. non-system dirs and not
defined in /etc/ld.so.conf.d/*)."

                
> User set java.library.path seems to overwrite default creating problems native lib loading
> ------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4072
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Anupam Seth
>            Assignee: Anupam Seth
>         Attachments: MAPREDUCE-4072-branch-23.patch, MAPREDUCE-4072-branch-23_documentation.patch
>
>
> This was found by Peeyush Bishnoi.
> While running a distributed cache example with Hadoop-0.23,
> tasks are failing as follows:
> ------------------------------------------------------------------------------------------------------------
> Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException: at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at
> org.apache.hadoop.util.Shell.run(Shell.java:188) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
> java.util.concurrent.FutureTask.run(FutureTask.java:138) at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user
> is <user>
> ------------------------------------------------------------------------------------------------------------
> Same Pig script and command work successfully on 0.20
> See this in the stderr:
> Exception in thread "main" java.lang.ExceptionInInitializerError
>     at java.lang.Class.forName0(Native Method)
>     at java.lang.Class.forName(Class.java:247)
>     at
> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179)
>     at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149)
>     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238)
>     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264)
>     at org.apache.hadoop.security.Groups.(Groups.java:54)
>     at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178)
>     at
> org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
>     at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
>     at
> org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265)
>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75)
> Caused by: java.lang.RuntimeException: Bailing out since native library
> couldn't be loaded
>     at
> org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48)
>     ... 12 more
> Pig command:
> $ pig -Dmapred.job.queue.name=<queue> -Dmapred.cache.archives=<archives>
-Dmapred.child.java.opts="-Djava.library.path=./ygeo/lib
> -Dip2geo.preLoadLibraries=<some other libs>" -Djava.io.tmpdir=/grid/0/tmp -Dmapred.create.symlink=yes
-Dmapred.job.map.memory.mb=3072 piggeoscript.pig

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message