hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rab ra <rab...@gmail.com>
Subject Re: simple hadoop MR program to be executed using java
Date Sat, 17 Jan 2015 16:13:31 GMT
On Sat, Jan 17, 2015 at 12:33 AM, Chris Nauroth <cnauroth@hortonworks.com>

> Hello Rab,
> There is actually quite a lot of logic in the "hadoop jar" shell scripts
> to set up the classpath (including Hadoop configuration file locations) and
> set up extra arguments (like heap sizes and log file locations).  It is
> possible to replicate it with a straight java call, but it might not be
> worth the effort, and end users of your jar would lose functionality
> implemented in the shell scripts, such as configuration file location
> overrides.
> If you still want to pursue this, then you might want to make a small
> change to the "hadoop jar" script and add a line right before the java call
> to echo the command it's running.  That will give you a sense for the java
> command that ultimately gets run.  You could also take a look at the
> process table for a running "hadoop jar" process and inspect its command
> line and environment variables.
> Another potentially helpful tool  is the "hadoop classpath" command:
> http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/CommandsManual.html#classpath
> This uses the full logic of the shell scripts for classpath construction,
> but then just echoes it instead of using it to run a jar.
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
> Hello,

Thanks for your response. I had a feeling that if a web application needs
to process a request from the client and subsequently span MR jobs, it
would not span command line process using 'hadoop' command and there would
be a way to instantiate a hadoop driver class that contains Mapper and
reducer. In this setup, I expected there would be a place where all the
hadoop related configuration / jars would be placed so that they are
available for hadoop job. Hence, asked this question. I thought it is
straightforward and many people would have attempted it and hence getting
help in the form of documentation and blog would not be problem. I spent
two days in this but still could not find a way to do this.

> On Fri, Jan 16, 2015 at 10:15 AM, rab ra <rabmdu@gmail.com> wrote:
>> Hello,
>> I have a simple java program that sets up a MR job. I could successfully
>> execute this in Hadoop infrastructure (hadoop 2x) using 'hadoop jar
>> <myjar>'. But I want to achieve the same thing using java command as below.
>> java <className>
>> 1. How can I pass hadoop configuration to this className?
>> 2. What extra arguments do I need to supply?
>> 3. Any link/documentation would be highly appreciated.
>> regards
>> rab
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

View raw message