hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Rosenstrauch <dar...@darose.net>
Subject Re: JobClient using deprecated JobConf
Date Mon, 27 Sep 2010 15:54:59 GMT
On 09/25/2010 10:24 AM, Martin Becker wrote:
> Hello David,
>
> thanks a lot. Yet I want java code to submit my application. I do not
> want to mess with any kind of command line arguments or an executable,
> neither Java nor Hadoop. I want to write a method that can set up and
> submit a job to an arbitrary cluster. Something like calling
> CustomJob.submitJob(ip:port). This would be used by a GUI or another
> java application to process data.

Also easy to do.

All that the -jt and -fs parms do is eventually set appropriate values 
in a Configuration object.  You can just as easily do this programmatically.

i.e., for -jt:

conf.set("mapred.job.tracker", 
<hostname_of_job_tracker_in_remote_cluster:job_tracker_port_number>);

for -fs:
conf.set("fs.default.name", 
<hdfs://hostname_of_name_node_in_remote_cluster:name_node_port_number>);


On a side note:  as far as your requirement "I do not want to mess with 
any kind of command line arguments or an executable, neither Java nor 
Hadoop", I'm not sure how feasible this requirement is because of the 
-libjars command line parm.

It's easy to write code to handle what the -fs and -jt command line 
parms do (see above).  But -libjars is much more complicated.  It takes 
the list of jars that you give it, sends each one to the Hadoop 
DistributedCache so that it gets distributed to each node in the 
cluster, and then adds each one to the classpath of each map/reduce tasks.

Yes, I suppose you could try to write code to do that yourself, but I 
can't see why you would want to reinvent the wheel here.  So this makes 
me question whether that requirement really makes sense.


> The problem is the the
> missing job.jar, as also described earlier. I will start a new thread
> describing my problem using a more accurate header.
>
> Thank you,
> Martin

I'm not sure what you're referring to re: job.jar.

DR

Mime
View raw message