hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brad Cox <bradj...@gmail.com>
Subject Re: Maven Cloudera Configuration problem
Date Tue, 13 Aug 2013 16:34:20 GMT
I've been stuck on the same question lately so don't take this as definitive, just my best
guess at what's required.

Using maven as your hadoop source is going to give you a "vanilla" hadoop; one that runs on
localhost. You need one that you've customized to point to your remote cluster and you can't
get that via maven. 

So my *GUESS* is you need to do a plain local install of hadoop and point HADOOP_HOME at that.
Customize as required, then convince eclipse to use that instead of going thru maven (i.e.
remove hadoop from the dependency list).

Everyone; is this on the right path? Anyone know of exact instructions?

On Aug 13, 2013, at 12:07 PM, Pavan Sudheendra <pavan0591@gmail.com> wrote:

> When i actually run the job on the multi node cluster, logs shows it
> uses localhost configurations which i don't want..
> I just have a pom.xml which lists all the dependencies like standard
> hadoop, standard hbase, standard zookeeper etc., Should i remove these
> dependencies?
> I want the cluster settings to apply in my map-reduce application..
> So, this is where i'm stuck at..
> On Tue, Aug 13, 2013 at 9:30 PM, Pavan Sudheendra <pavan0591@gmail.com> wrote:
>> Hi Shabab and Sandy,
>> The thing is we have a 6 node cloudera cluster running.. For
>> development purposes, i was building a map-reduce application on a
>> single node apache distribution hadoop with maven..
>> To be frank, i don't know how to deploy this application on a multi
>> node cloudera cluster. I am fairly well versed with Multi Node Apache
>> Hadoop Distribution.. So, how can i go forward?
>> Thanks for all the help :)
>> On Tue, Aug 13, 2013 at 9:22 PM,  <sandy.ryza@cloudera.com> wrote:
>>> Hi Pavan,
>>> Configuration properties generally aren't included in the jar itself unless you
explicitly set them in your java code. Rather they're picked up from the mapred-site.xml file
located in the Hadoop configuration directory on the host you're running your job from.
>>> Is there an issue you're coming up against when trying to run your job on a cluster?
>>> -Sandy
>>> (iphnoe tpying)
>>> On Aug 13, 2013, at 4:19 AM, Pavan Sudheendra <pavan0591@gmail.com> wrote:
>>>> Hi,
>>>> I'm currently using maven to build the jars necessary for my
>>>> map-reduce program to run and it works for a single node cluster..
>>>> For a multi node cluster, how do i specify my map-reduce program to
>>>> ingest the cluster settings instead of localhost settings?
>>>> I don't know how to specify this using maven to build my jar.
>>>> I'm using the cdh distribution by the way..
>>>> --
>>>> Regards-
>>>> Pavan
>> --
>> Regards-
>> Pavan
> -- 
> Regards-
> Pavan

Dr. Brad J. Cox    Cell: 703-594-1883 Blog: http://bradjcox.blogspot.com http://virtualschool.edu

View raw message