hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Rosenstrauch <dar...@darose.net>
Subject Re: Deployment of jar files at startup
Date Wed, 15 Sep 2010 17:46:19 GMT
On 09/14/2010 10:10 PM, Pete Tyler wrote:
> I'm trying to figure out how to achieve the following from a Java client,
> 1. My app (which is a web server) starts up
> 2. As part of startup my jar file, which includes my map reduce classes are distributed
to hadoop nodes
> 3. My web app uses map reduce to extract data without the performance overhead of each
job deploying a jar file, via setJar(), setJarByClass()
>
> It looks like DistributedCache() has potential but the need for commands like 'hadoop
fs -copyFromLocal ...' and the API methods like '.getLocalCacheArchives()' look to be at odds
with my scenario. Any thoughts?
>
> -Peter

For step 2, you have 2 options on how to implement:
a) call DistributedCache.addFileToClassPath(jarFileURI, conf);
b) have your app implement Tool, use ToolRunner to launch it, and 
specify a -libjars command line parm which will achieve the same effect 
as in (a).  See 
http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/util/Tool.html 
and 
http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/util/GenericOptionsParser.html#GenericOptions

for details.

HTH,

DR

Mime
View raw message