hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Blah <tmp5...@gmail.com>
Subject Re: Implementing and running an applicationmaster
Date Thu, 05 Dec 2013 09:28:06 GMT
Hi

If I understood you correctly, you would like to run your AM with YARN
Client from shell as oppose to run the Driver like in MRv1. But it's the
same thing (more or less). In the example you provided
(org.apache.hadoop.yarn.applications.DistributedShell) the Client.class is
the "driver". However since distributed-shell is a "simple" application you
do not need a lot of configuration (setting fields in Configuration.class,
I/O formats etc.). The same goes for any other application. As for the
second example (org.apache.hadoop.examples.WordCount) MapReduce AM requires
certain configuration, thus you have to to it the "old-way". The main
difference would be: MR -> end-user-config -> driver, DS -> driver (but you
still can create your own end-user-config). Hope this answers your question
and that I understood it correctly.

regards
tmp


2013/12/5 Yue Wang <terranwy@gmail.com>

> Hi,
>
> I took a look at the codes and found some examples on the web.
> One example is: http://wiki.opf-labs.org/display/SP/Resource+management
>
> It seems that users can run simple shell commands using Client of YARN.
> But when it comes to a practical MapReduce example like WordCount, people
> still run commands in the old way as in MRv1.
>
> How can I run WordCount using Client and ApplicationMaster of YARN so that
> I can request resources flexibly?
>
>
> Thanks!
>
>
> On Mon, Dec 2, 2013 at 11:26 AM, Rob Blah <tmp5330@gmail.com> wrote:
>
>> Hi
>>
>> Follow the example provided in
>> Yarn_dist/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
>>
>> regards
>> tmp
>>
>>
>> 2013/12/1 Yue Wang <terranwy@gmail.com>
>>
>>> Hi,
>>>
>>> I found the page (
>>> http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html)
>>> and know how to write an ApplicationMaster.
>>>
>>> However, is there a complete example showing how to run this
>>> ApplicationMaster with a real Hadoop Program (e.g. WordCount) on YARN?
>>>
>>> Thanks!
>>>
>>>
>>>
>>> Yue
>>>
>>
>>
>

Mime
View raw message