hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yaron Gonen <yaron.go...@gmail.com>
Subject Re: Submitting a Hadoop task from withing a reducer
Date Wed, 05 Oct 2011 13:50:52 GMT
Thanks guys. I'll dig a little in Yarn.

On Tue, Oct 4, 2011 at 6:28 PM, Joey Echeverria <joey@cloudera.com> wrote:

> Yes. The reason I pointed it to him was that it seems like he's trying
> to do something with Hadoop for which MapReduce may not be right
> execution model. Yarn/MRv2 gives you the ability to try other
> execution models. As you pointed out, it may require some extra
> development, but it is more flexible than straight MapReduce and
> probably a better option in the long run over submitting jobs from
> inside of jobs.
>
> -Joey
>
> On Tue, Oct 4, 2011 at 12:11 PM, GOEKE, MATTHEW (AG/1000)
> <matthew.goeke@monsanto.com> wrote:
> > Joey,
> >
> > Is yarn just a synonym for MRv2? And if so he would still have to create
> a custom application master for his job type right?
> >
> > Matt
> >
> > -----Original Message-----
> > From: Joey Echeverria [mailto:joey@cloudera.com]
> > Sent: Tuesday, October 04, 2011 11:06 AM
> > To: mapreduce-user@hadoop.apache.org
> > Subject: Re: Submitting a Hadoop task from withing a reducer
> >
> > You may want to check out Yarn, coming in Hadoop 0.23:
> >
> > https://issues.apache.org/jira/browse/MAPREDUCE-279
> >
> > -Joey
> >
> > On Tue, Oct 4, 2011 at 11:45 AM, Yaron Gonen <yaron.gonen@gmail.com>
> wrote:
> >> Hi,
> >> Hadoop tasks are always stacked to form a linear user-managed workflow
> (a
> >> reduce step cannot start before all previous mappers have stopped etc).
> This
> >> may be problematic in recursive tasks: for example in a BFS we will not
> get
> >> any output until the longest branch has been reached.
> >> In order to solve than, an idea came up of submitting a whole Hadoop
> task
> >> from within a reducer. Have anyone tried it?
> >> Thanks.
> >>
> >
> >
> >
> > --
> > Joseph Echeverria
> > Cloudera, Inc.
> > 443.305.9434
> > This e-mail message may contain privileged and/or confidential
> information, and is intended to be received only by persons entitled
> > to receive such information. If you have received this e-mail in error,
> please notify the sender immediately. Please delete it and
> > all attachments from any servers, hard drives or any other media. Other
> use of this e-mail by you is strictly prohibited.
> >
> > All e-mails and attachments sent and received are subject to monitoring,
> reading and archival by Monsanto, including its
> > subsidiaries. The recipient of this e-mail is solely responsible for
> checking for the presence of "Viruses" or other "Malware".
> > Monsanto, along with its subsidiaries, accepts no liability for any
> damage caused by any such code transmitted by or accompanying
> > this e-mail or any attachment.
> >
> >
> > The information contained in this email may be subject to the export
> control laws and regulations of the United States, potentially
> > including but not limited to the Export Administration Regulations (EAR)
> and sanctions regulations issued by the U.S. Department of
> > Treasury, Office of Foreign Asset Controls (OFAC).  As a recipient of
> this information you are obligated to comply with all
> > applicable U.S. export laws and regulations.
> >
> >
>
>
>
> --
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434
>

Mime
View raw message