hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Roman Shaposhnik (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3693) Job's shouldn't have to configure the location of compression libraries on the server in mapred-site.xml
Date Thu, 19 Jan 2012 22:22:39 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189428#comment-13189428
] 

Roman Shaposhnik commented on MAPREDUCE-3693:
---------------------------------------------

Now that I've looked at the actual implementation (Arun, thanks a lot for pointing out where
it is!) the only concern that I have left is that it is *extremely* difficult to realize that
mapreduce.admin.user.env is what actually adds 
LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native to the YarnChild's environment. 

In other words, if the end user wants to tweak  mapreduce.admin.user.env it is a hard requirement
to NOT forget adding LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native back to whatever user
wants to pass via that property. This strikes me as a pretty brittle implementation, unless
of course mapreduce.admin.user.env is meant to be one of those hidden options that
you're better off not tweaking.

In summary, I really do think that one of the following needs to happen:
   * $HADOOP_COMMON_HOME/lib/native needs to be appended to whatever value of LD_LIBRARY_PATH
there is in all cases
   * mapreduce.admin.user.env needs to be documents as a do not touch option
   * mapreduce.admin.user.env needs to be documents with a NOTE saying that setting it will
invalidate the default setting of LD_LIBRARY_PATH (which will break loading of your native
libraries)
                
> Job's shouldn't have to configure the location of compression libraries on the server
in mapred-site.xml
> --------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3693
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3693
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Roman Shaposhnik
>
> I have noticed that org.apache.hadoop.mapred.MapReduceChildJVM doesn't forward the value
of -Djava.library.path= from the parent JVM to the child JVM. Thus if one wants to use native
libraries for compression the only option seems to be to manually include relevant java.library.path
settings into the mapred-site.xml (as mapred.[map|reduce].child.java.opts).
> This seems to be a change in behavior compared to MR1 where TaskRunner.java used to do
that:
> {noformat}
> String libraryPath = System.getProperty("java.library.path");
>     if (libraryPath == null) {
>       libraryPath = workDir.getAbsolutePath();
>     } else {
>       libraryPath += SYSTEM_PATH_SEPARATOR + workDir;
>     }
>     boolean hasUserLDPath = false;
>     for(int i=0; i<javaOptsSplit.length ;i++) {
>       if(javaOptsSplit[i].startsWith("-Djava.library.path=")) {
>         javaOptsSplit[i] += SYSTEM_PATH_SEPARATOR + libraryPath;
>         hasUserLDPath = true;
>         break;
>       }
>     }
>     if(!hasUserLDPath) {
>       vargs.add("-Djava.library.path=" + libraryPath);
>     }
>     for (int i = 0; i < javaOptsSplit.length; i++) {
>       vargs.add(javaOptsSplit[i]);
>     }
> {noformat}
> Is this a regression or a deliberate choice?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message