hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Schubert Zhang <zson...@gmail.com>
Subject Re: Optimized Hadoop
Date Fri, 17 Feb 2012 04:25:53 GMT
1) it should be sort-avoidance.
2) work pool (like Tenzing)

Sorry ,the adaptive heartbeat code is not in this github code, we are
discussing it.


On Fri, Feb 17, 2012 at 11:00 AM, Anty <anty.rao@gmail.com> wrote:

> Hi: Todd
>
> yes, the rewritten shuffle in actual a backport of the shuffle from MR2 .
> We mainly add the following two features:
> 1) shuffle avoidance
> 2) work pool
>
>
> On Fri, Feb 17, 2012 at 3:27 AM, Todd Lipcon <todd@cloudera.com> wrote:
>
>> Hey Schubert,
>>
>> Looking at the code on github, it looks like your rewritten shuffle is
>> in fact just a backport of the shuffle from MR2. I didn't look closely
>> - are there any distinguishing factors?
>> Also, the OOB heartbeat and adaptive heartbeat code seems to be the
>> same as what's in 1.0?
>>
>> -Todd
>>
>> On Thu, Feb 16, 2012 at 9:44 AM, Schubert Zhang <zsongbo@gmail.com>
>> wrote:
>> > Here is the presentation to describe our job,
>> >
>> http://www.slideshare.net/hanborq/hanborq-optimizations-on-hadoop-mapreduce-20120216a
>> > Wellcome to give your advises.
>> > It's just a little step, and we are continue to do more improvements,
>> thanks
>> > for your help.
>> >
>> >
>> >
>> >
>> > On Thu, Feb 16, 2012 at 11:01 PM, Anty <anty.rao@gmail.com> wrote:
>> >>
>> >> Hi: Guys
>> >>        We just deliver a optimized hadoop , if you are interested, Pls
>> >> refer to https://github.com/hanborq/hadoop
>> >>
>> >> --
>> >> Best Regards
>> >> Anty Rao
>> >
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Best Regards
> Anty Rao
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message