hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 顾荣 <gurongwal...@gmail.com>
Subject Re: Shared lib?
Date Mon, 11 Apr 2011 15:11:40 GMT
Hi Mark,
 I also met your problem,I found my way finally.
 Firstly,your basic idea is right,we need to move these jars in to
HDFS,because files in HDFS are shared by all the node automatically.
 So,There seem to be two solutions here.

 solution: a)After you export your project as a jar,you add a directory
named lib in the root of your project jar manually,after that you
  can add ALL external jars which your project needs in this directory made
by you. By this way,when you submit your job to the jobtracker,
  you also submit the lib jars into the HDFS together,and Hadoop will add
your small lib into its lib for your project.I tried,it works.This may
  reduce your work,I think.

 solution: b)There seems to be  a Hadoop command with parameter as '-libjars
jar1,jar2' ,I read this from *Hadoop.The.Definitive.Guide.*
* *the description of this command is*"*Copies the specified JAR files from
the local filesystem (or any filesystem if a scheme is specified)    to the
shared filesystem used by the jobtracker (usually HDFS), and adds them to
the MapReduce task’s classpath. This option is a  useful way of shipping JAR
files that *a job is dependent on." *howerver,I still did not figure it out
now,It did not works for me.

The above are what I have experienced,I hope it can help you.
 If you still have some problems,please feel free to let me.Good Luck.

Walker Gu.

2011/4/9 Mark <static.void.dev@gmail.com>

> If I have some jars that I would like including into ALL of my jobs is
> there some shared lib directory on HDFS that will be available to all of my
> nodes? Something similar to how Oozie uses a shared lib directory when
> submitting workflows.
> As of right now I've been cheating and copying these jars into the hadoop
> lib directory on each node. I figured there had to be a better way.
> Thanks

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message