hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "GOEKE, MATTHEW (AG/1000)" <matthew.go...@monsanto.com>
Subject RE: Submitting a Hadoop task from withing a reducer
Date Tue, 04 Oct 2011 16:11:00 GMT
Joey,

Is yarn just a synonym for MRv2? And if so he would still have to create a custom application
master for his job type right?

Matt

-----Original Message-----
From: Joey Echeverria [mailto:joey@cloudera.com] 
Sent: Tuesday, October 04, 2011 11:06 AM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Submitting a Hadoop task from withing a reducer

You may want to check out Yarn, coming in Hadoop 0.23:

https://issues.apache.org/jira/browse/MAPREDUCE-279

-Joey

On Tue, Oct 4, 2011 at 11:45 AM, Yaron Gonen <yaron.gonen@gmail.com> wrote:
> Hi,
> Hadoop tasks are always stacked to form a linear user-managed workflow (a
> reduce step cannot start before all previous mappers have stopped etc). This
> may be problematic in recursive tasks: for example in a BFS we will not get
> any output until the longest branch has been reached.
> In order to solve than, an idea came up of submitting a whole Hadoop task
> from within a reducer. Have anyone tried it?
> Thanks.
>



-- 
Joseph Echeverria
Cloudera, Inc.
443.305.9434
This e-mail message may contain privileged and/or confidential information, and is intended
to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the
sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail
by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, reading and archival
by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence
of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such
code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export control laws and regulations
of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations
issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC).  As a recipient of this information you
are obligated to comply with all
applicable U.S. export laws and regulations.


Mime
View raw message