Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1C0EB9726 for ; Mon, 20 Feb 2012 18:18:27 +0000 (UTC) Received: (qmail 98975 invoked by uid 500); 20 Feb 2012 18:18:19 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 98730 invoked by uid 500); 20 Feb 2012 18:18:19 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 98695 invoked by uid 99); 20 Feb 2012 18:18:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Feb 2012 18:18:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of zsongbo@gmail.com designates 209.85.214.176 as permitted sender) Received: from [209.85.214.176] (HELO mail-tul01m020-f176.google.com) (209.85.214.176) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Feb 2012 18:18:12 +0000 Received: by obbwd18 with SMTP id wd18so10382330obb.35 for ; Mon, 20 Feb 2012 10:17:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=nJ1ETzg/Nq0KzeYF5oIMqcNtZHjBuC+iiqQKGWqLevc=; b=MX5Oj7RRam90s2xtxCJbI9IajHTS2a9BxFZokOMs9a14iZtl7izo8TcRpkGRmkMBj1 ZcSU5E1BW8L7cyH0tQwoChUXFYIi/6JvlQshJN7svy7eIFs4DWa1pwe0sFQHlUcogMQn 80rVTsTE1ExbU2o9Yzvdikuzfu4EZoSIzncnw= Received: by 10.182.75.102 with SMTP id b6mr11943377obw.9.1329761871130; Mon, 20 Feb 2012 10:17:51 -0800 (PST) MIME-Version: 1.0 Received: by 10.60.33.162 with HTTP; Mon, 20 Feb 2012 10:17:31 -0800 (PST) In-Reply-To: References: From: Schubert Zhang Date: Tue, 21 Feb 2012 02:17:31 +0800 Message-ID: Subject: Re: Optimized Hadoop To: user@hive.apache.org Cc: mapreduce-user@hadoop.apache.org, mapreduce-dev@hadoop.apache.org, hive-user@hadoop.apache.org, common-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=14dae93994a9f6ef8b04b9694fca X-Virus-Checked: Checked by ClamAV on apache.org --14dae93994a9f6ef8b04b9694fca Content-Type: text/plain; charset=ISO-8859-1 We just update the slides of this improvements: http://www.slideshare.net/hanborq/hanborq-optimizations-on-hadoop-mapreduce-20120216a Updates: (1) modified some describes to make things more clear and accuracy. (2) add some benchmarks to make sense. On Sat, Feb 18, 2012 at 11:12 PM, Anty wrote: > > > On Fri, Feb 17, 2012 at 3:27 AM, Todd Lipcon wrote: > >> Hey Schubert, >> >> Looking at the code on github, it looks like your rewritten shuffle is >> in fact just a backport of the shuffle from MR2. I didn't look closely >> > > additionally, the rewritten shuffle in MR2 has some bugs, which harm the > overall performance, for which I have already file a jira to report this, > with a patch available. > MAPREDUCE-3685 > > > >> - are there any distinguishing factors? >> Also, the OOB heartbeat and adaptive heartbeat code seems to be the >> same as what's in 1.0? >> >> -Todd >> >> On Thu, Feb 16, 2012 at 9:44 AM, Schubert Zhang >> wrote: >> > Here is the presentation to describe our job, >> > >> http://www.slideshare.net/hanborq/hanborq-optimizations-on-hadoop-mapreduce-20120216a >> > Wellcome to give your advises. >> > It's just a little step, and we are continue to do more improvements, >> thanks >> > for your help. >> > >> > >> > >> > >> > On Thu, Feb 16, 2012 at 11:01 PM, Anty wrote: >> >> >> >> Hi: Guys >> >> We just deliver a optimized hadoop , if you are interested, Pls >> >> refer to https://github.com/hanborq/hadoop >> >> >> >> -- >> >> Best Regards >> >> Anty Rao >> > >> > >> >> >> >> -- >> Todd Lipcon >> Software Engineer, Cloudera >> > > > > -- > Best Regards > Anty Rao > --14dae93994a9f6ef8b04b9694fca Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable We just update the slides of this improvements:
h= ttp://www.slideshare.net/hanborq/hanborq-optimizations-on-hadoop-mapreduce-= 20120216a

Updates:
(1) modified some describes to make things more clear and a= ccuracy.
(2) add some benchmarks to make sense.

On Sat, Feb 18, 2012 at 11:12 PM, Anty <anty.rao@gmail.com> wrote:<= br>


On Fri, Feb 17, 2012 at 3:27 AM, Todd Lipcon <= ;todd@cloudera.com> wrote:
Hey Schubert,

Looking at the code on github, it looks like your rewritten shuffle is
in fact just a backport of the shuffle from MR2. I didn't look closely<= br>
- are there any distinguishing factors?
Also, the OOB heartbeat and adaptive heartbeat code seems to be the
same as what's in 1.0?

-Todd

On Thu, Feb 16, 2012 at 9:44 AM, Schubert Zhang <zsongbo@gmail.com> wrote:
> Here is the presentation to describe our job,
> http://www.slideshare.net/han= borq/hanborq-optimizations-on-hadoop-mapreduce-20120216a
> Wellcome to give your advises.
> It's just a little step, and we are continue to do more improvemen= ts, thanks
> for your help.
>
>
>
>
> On Thu, Feb 16, 2012 at 11:01 PM, Anty <anty.rao@gmail.com> wrote:
>>
>> Hi: Guys
>> =A0=A0=A0=A0=A0=A0 We just deliver a optimized hadoop , if you are= interested, Pls
>> refer to https://github.com/hanborq/hadoop
>>
>> --
>> Best Regards
>> Anty Rao
>
>



--
Todd Lipcon
Software Engineer, Cloudera



--
Best Regards
Anty Rao

--14dae93994a9f6ef8b04b9694fca--