hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joey Echeverria <j...@cloudera.com>
Subject Re: Submitting a Hadoop task from withing a reducer
Date Tue, 04 Oct 2011 16:28:57 GMT
Yes. The reason I pointed it to him was that it seems like he's trying
to do something with Hadoop for which MapReduce may not be right
execution model. Yarn/MRv2 gives you the ability to try other
execution models. As you pointed out, it may require some extra
development, but it is more flexible than straight MapReduce and
probably a better option in the long run over submitting jobs from
inside of jobs.


On Tue, Oct 4, 2011 at 12:11 PM, GOEKE, MATTHEW (AG/1000)
<matthew.goeke@monsanto.com> wrote:
> Joey,
> Is yarn just a synonym for MRv2? And if so he would still have to create a custom application
master for his job type right?
> Matt
> -----Original Message-----
> From: Joey Echeverria [mailto:joey@cloudera.com]
> Sent: Tuesday, October 04, 2011 11:06 AM
> To: mapreduce-user@hadoop.apache.org
> Subject: Re: Submitting a Hadoop task from withing a reducer
> You may want to check out Yarn, coming in Hadoop 0.23:
> https://issues.apache.org/jira/browse/MAPREDUCE-279
> -Joey
> On Tue, Oct 4, 2011 at 11:45 AM, Yaron Gonen <yaron.gonen@gmail.com> wrote:
>> Hi,
>> Hadoop tasks are always stacked to form a linear user-managed workflow (a
>> reduce step cannot start before all previous mappers have stopped etc). This
>> may be problematic in recursive tasks: for example in a BFS we will not get
>> any output until the longest branch has been reached.
>> In order to solve than, an idea came up of submitting a whole Hadoop task
>> from within a reducer. Have anyone tried it?
>> Thanks.
> --
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434
> This e-mail message may contain privileged and/or confidential information, and is intended
to be received only by persons entitled
> to receive such information. If you have received this e-mail in error, please notify
the sender immediately. Please delete it and
> all attachments from any servers, hard drives or any other media. Other use of this e-mail
by you is strictly prohibited.
> All e-mails and attachments sent and received are subject to monitoring, reading and
archival by Monsanto, including its
> subsidiaries. The recipient of this e-mail is solely responsible for checking for the
presence of "Viruses" or other "Malware".
> Monsanto, along with its subsidiaries, accepts no liability for any damage caused by
any such code transmitted by or accompanying
> this e-mail or any attachment.
> The information contained in this email may be subject to the export control laws and
regulations of the United States, potentially
> including but not limited to the Export Administration Regulations (EAR) and sanctions
regulations issued by the U.S. Department of
> Treasury, Office of Foreign Asset Controls (OFAC).  As a recipient of this information
you are obligated to comply with all
> applicable U.S. export laws and regulations.

Joseph Echeverria
Cloudera, Inc.

View raw message