hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: submitting jobs from a webapp
Date Mon, 31 Jan 2011 17:49:07 GMT
(moving this to the user mailing list where it belongs)

You need to make sure that your webapp knows the address of the
JobTracker, usually this is done by either putting mapred-site.xml on
your app's classpath or you can set mapred.job.tracker correctly so
that in createSubmittableJob you would have:

conf.set("mapred.job.tracker", "whatever the address is:and the port");

Else it will run in local mode.

J-D

On Mon, Jan 31, 2011 at 7:09 AM,  <eric_bdr@yahoo.com> wrote:
> Hi,
>
> Sorry if this is a trivial question but I have been battling with it for a
> while.  I have a question about submitting jobs from a web application
> potentially deployed on different machines than HBase.  I am using
> springframework and implemented a simple JobService class that acts as a spring
> service.  I can then implement all my jobs in separate classes and pass them to
> my jobService for execution.  When I run jobs through the hadoop command line,
> the job shows up in the job tracker.  However, when I submit jobs through my
> webapp, it does not.  Since I can step through the execution of my job in my
> eclipse environment, I don't think this job is running in hadoop/hbase's
> distributed environment.  Is that right?
>
> 1. In short, how can I submit a job from my webapp and view its progress on the
> jobtracker?
>
> 2. Additionally, I would also like to be able to get this information to display
> it in my own web application (so that users can track the status of their
> running jobs).  How can I query the Hadoop job tracker?
>
> Thanks a lot for your help,
> -Eric
>
> here is snipets of my code:
>
> class myJob extends IJob {
> ...
>   public Job createSubmittableJob(Configuration conf, String[] args)
> Path inputDir = new Path(inputDir);
> job = new Job(conf, jobName);
> job.setJarByClass(Uploader.class);
> FileInputFormat.setInputPaths(job, inputDir);
> job.setInputFormatClass(TextInputFormat.class);
> job.setMapperClass(TransactionLoader.class);
> TableMapReduceUtil.initTableReducerJob(tableName, null, job);
> job.setNumReduceTasks(0);
>    }
> ...
> }
>
> class myJobService {
> ...
>
>   public int submitJob(String name, IJob zjob, String[] args) {
>
> job = zjob.createSubmittableJob(dsm.getHBaseConfiguration(), def, otherArgs);
> // "Submit the job to the cluster and wait for it to finish."
> return job.waitForCompletion(true) ? 0 : 1;
>   }
> ...
> }

Mime
View raw message