hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henning Blohm <henning.bl...@zfabrik.de>
Subject Setting extra classpath and library path for Yarn M/R Application
Date Wed, 30 Mar 2016 10:54:35 GMT
Hi,

I am having a hard time figuring out how to add command line options, an 
extra class path and library path to a map reduce application running on 
Hadoop 2.6.0 (CDH 5.6).

Unfortunately this application is using a native library with a 
pre-installed wrapper JAR that is installed on all data nodes. So what I 
am trying to achieve is to pass an LD_LIBRARY_PATH and an extra 
classpath to the application runner.

Previously, I used

     <property>
         <name>mapred.child.java.opts</name>
         <value>-XX:-UseSplitVerifier -Xmx500M -XX:MaxPermSize=256M 
-Dnosql.master=${nosql.master}</value>
     </property>

in the client config file for command line options and set

export HADOOP_CLASSPATH=/pathToWrapperJar
export JAVA_LIBRARY_PATH=/nativepath/

in my Hadoop environment.  None of this seems to work with Hadoop 2.6.0 
(Yarn).

Here is what I have now:

I managed to add to LD_LIBRARY_PATH by specifying

     <property>
         <name>yarn.app.mapreduce.am.env</name>
         <value>LD_LIBRARY_PATH=/nativepath</value>
     </property>

     <property>
         <name>yarn.app.mapreduce.am.command-opts</name>
         <value>-Da=1 -Db=2</value>
     </property>

in the client config file. I can tell that by cat'ing the 
launch_container.sh while it is present.

I do however fail to add anything to the application's class path. Plus 
I would rather not specify the native library path and wrapper jar in 
the client config but rather globally.

It would be great, if somebody can give me a hint first and foremost on 
the class path question and secondly some orientation on whether using 
the new configuration properties is the way to go.

Thanks,
Henning


Mime
View raw message