hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rab ra <rab...@gmail.com>
Subject Launching Hadoop map reduce job from a servlet
Date Fri, 16 Jan 2015 17:53:09 GMT
Hello,

I have a servlet program deployed in jetty server listening to the port
8080. As soon as a request arrives from a client, it parses the request,
and instantiate MR program that is to be launched in Hadoop cluster. Here,
I cannot launch the hadoop job using hadoop command as 'hadoop jar <jar
file> .....". From servlet code, I instantiate the MR main program that
implements Tools and contains Mapper and Reducer classes in it.

My issue is that though the job is launched, it always uses Local
JobRunner. I do have hadoop installed with all the configuration files
contains right information in it. For instance, in my mapred-site.xml, i
have setup 'yarn' as my mapreduce framework.

With the current configuration setup, I was able to submit jobs to yarn
through hadoop command. But i want to achieve this through 'java' command.

1. How can I do it? If there is any pointer/link, please share it.
2. I tried to setup all the configuration inside the code something like as
below
....
conf.set("mapreduce,framework.name","yarn");
....
....

But somehow, it seems that these information is not cascading to job
despite creating job instance with the above configuration. So, I am
struggling to make hadoop configuration to java application.

I would be grateful to you for any help to fix this issue


regards
rab

Mime
View raw message