hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4072) User set java.library.path seems to overwrite default creating problems native lib loading
Date Tue, 03 Apr 2012 18:06:24 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Robert Joseph Evans updated MAPREDUCE-4072:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 2.0.0
                   0.23.3
     Release Note: -Djava.library.path in mapred.child.java.opts can cause issues with native
libraries.  LD_LIBRARY_PATH through mapred.child.env should be used instead.
     Hadoop Flags: Incompatible change,Reviewed
           Status: Resolved  (was: Patch Available)

Thanks Anupam.  +1 on the documentation.  I have checked this into trunk, branch-2, and branch-0.23
                
> User set java.library.path seems to overwrite default creating problems native lib loading
> ------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4072
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Anupam Seth
>            Assignee: Anupam Seth
>             Fix For: 0.23.3, 2.0.0
>
>         Attachments: MAPREDUCE-4072-branch-23.patch, MAPREDUCE-4072-branch-23_documentation.patch,
MAPREDUCE-4072-branch-23_documentation.patch
>
>
> This was found by Peeyush Bishnoi.
> While running a distributed cache example with Hadoop-0.23,
> tasks are failing as follows:
> ------------------------------------------------------------------------------------------------------------
> Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException: at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at
> org.apache.hadoop.util.Shell.run(Shell.java:188) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
> java.util.concurrent.FutureTask.run(FutureTask.java:138) at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user
> is <user>
> ------------------------------------------------------------------------------------------------------------
> Same Pig script and command work successfully on 0.20
> See this in the stderr:
> Exception in thread "main" java.lang.ExceptionInInitializerError
>     at java.lang.Class.forName0(Native Method)
>     at java.lang.Class.forName(Class.java:247)
>     at
> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179)
>     at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149)
>     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238)
>     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264)
>     at org.apache.hadoop.security.Groups.(Groups.java:54)
>     at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178)
>     at
> org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
>     at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
>     at
> org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265)
>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75)
> Caused by: java.lang.RuntimeException: Bailing out since native library
> couldn't be loaded
>     at
> org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48)
>     ... 12 more
> Pig command:
> $ pig -Dmapred.job.queue.name=<queue> -Dmapred.cache.archives=<archives>
-Dmapred.child.java.opts="-Djava.library.path=./ygeo/lib
> -Dip2geo.preLoadLibraries=<some other libs>" -Djava.io.tmpdir=/grid/0/tmp -Dmapred.create.symlink=yes
-Dmapred.job.map.memory.mb=3072 piggeoscript.pig

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message