hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Something Something <mailinglist...@gmail.com>
Subject Re: Distributing our jars to all machines in a cluster
Date Wed, 16 Nov 2011 06:39:50 GMT
Until now we were manually copying our Jars to all machines in a Hadoop
cluster.  This used to work until our cluster size was small.  Now our
cluster is getting bigger.  What's the best way to start a Hadoop Job that
automatically distributes the Jar to all machines in a cluster?

I read the doc at:
http://hadoop.apache.org/common/docs/current/commands_manual.html#jar

Would -libjars do the trick?  But we need to use 'hadoop job' for that,
right?  Until now, we were using 'hadoop jar' to start all our jobs.

Needless to say, we are getting our feet wet with Hadoop, so appreciate
your help with our dumb questions.

Thanks.

PS:  We use Pig a lot, which automatically does this, so there must be a
clean way to do this.

Mime
View raw message