hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bertrand Dechoux <decho...@gmail.com>
Subject Re: mr1 and mr2
Date Thu, 15 May 2014 12:33:19 GMT
Or basically any other mature tools (even though I would also recommend
Cascading). The migration should not be the responsibility of the
application. If it is, it means that every application needs to have
(useless) glue code...

Bertrand


On Wed, May 14, 2014 at 12:31 PM, Andre Kelpe <akelpe@concurrentinc.com>wrote:

> May I recommend using Cascading instead of using MR directly? Cascading
> supports Hadoop 1.x and Hadoop 2.x based distros and you don't have to
> wrestle with these things all the time: http://www.cascading.org/ It's
> OSS, ASL v2 licensed and all the good stuff.
>
> - André
>
>
> On Sun, May 11, 2014 at 1:52 AM, Tony Dean <Tony.Dean@sas.com> wrote:
>
>>  Hi,
>>
>>
>>
>> I am trying to write a Java application that works with either MR1 and
>> MR2.  At the present I have MR2 (YARN) implementation deployed and
>> running.  I am using mapred API.  I believe that I read mapred and
>> mapreduce APIs are compatible so either should work.  The only thing that
>> is different is the configuration properties that need to be specified
>> depending on whether the back-end is MR1 or MR2. BTW: I’m using CDH 4.6
>> (Hadoop 2.0).
>>
>>
>>
>> My problem is that I can’t seem to submit a job to the cluster.  It
>> always runs locally.  I setup JobConf with appropriate properties and
>> submit the jobs using JobClient.  The properties that I set on JobConf are
>> as follows:
>>
>>
>>
>> mapreduce.jobtracker.address=host:port (I know this is for MR1, but I’m
>> trying everything)
>>
>> mapreduce.framework.name=yarn
>>
>> yarn.resourcemanager.address=host:port
>>
>> yarn.resourcemanager.host=host:port
>>
>>
>>
>> The last 2 are the same but I read 2 different ways to set it in
>> different conflicting documentations.
>>
>>
>>
>> Anyway, can someone explain how to get this seemingly simple deployment
>> to work?  What am I missing?
>>
>>
>>
>> Thanks!!!
>>
>
>
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
>

Mime
View raw message