Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of boyuzhang35@gmail.com
 designates 74.125.82.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jd/X2iijz20Rd7X1DuvUeI0+IZKJ12++1Y0pbhfKRihqkUWteVZyzf+XSxctcH3XA1
         DAHkx5euvyarT0Y8OnyKpMpCxhIHYhH/riGladJoKmj8ksUeNueb/8yHLlABWuYahp2/
         Y7rP+IS3/KiTaRSwbOux47sLBG5uO28AY+pug=
MIME-Version: 1.0
In-Reply-To: <AANLkTinejAaa2+PuxMoTHfH07H2NTDhmCZVvHnjnFsVt@mail.gmail.com>
References: <AANLkTim0iCNwX7OrYF+Rni4ph4iQg9u5K+nMYWFk7E0V@mail.gmail.com>
	<AANLkTi=nj_9r-thUuLBGLpYFmfSe0m8Yswq8tqqQy6nS@mail.gmail.com>
	<AANLkTi=5rLR7dd4JakGJnk9zAkJQ=y4kvX9LQT_pmVDx@mail.gmail.com>
	<AANLkTinrO74W8-Ev-TNqbS6CNSf2yQK-MpOKf0TU-EiM@mail.gmail.com>
	<AANLkTikRMcZ8pmWCdtvL=apaRe3X0OKT2iMpafD6T_Qe@mail.gmail.com>
	<AANLkTinejAaa2+PuxMoTHfH07H2NTDhmCZVvHnjnFsVt@mail.gmail.com>
Date: Fri, 13 Aug 2010 16:21:03 -0400
Message-ID: <AANLkTikaENrJ_mj5hnEsvHwkv75ntps-gcxiQZxWu3u8@mail.gmail.com>
Subject: Re: large parameter file, too many intermediate output
From: Boyu Zhang <boyuzhang35@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151750e3b8ccf9f3048dba3872

--00151750e3b8ccf9f3048dba3872
Content-Type: text/plain; charset=ISO-8859-1

Hi Harsh,

Thank you for the reply. I will try that, although now the map tasks are
taking too much time, almost 20 min to finish all the map tasks(~90). I
don't know if compression will slow me down, but I will make a test and see.
Thank you very much!

Boyu

On Thu, Aug 12, 2010 at 11:07 PM, Harsh J <qwertymaniac@gmail.com> wrote:

> Apart from the combiner suggestion, I'd also suggest using
> intermediate map-output compression always (With LZO, if possible).
> Saves you some IO.
>
> On Fri, Aug 13, 2010 at 3:24 AM, Boyu Zhang <boyuzhang35@gmail.com> wrote:
> > Hi Steve,
> >
> > Thanks for the reply!
> >
> > On Thu, Aug 12, 2010 at 5:47 PM, Steve Lewis <lordjoe2000@gmail.com>
> wrote:
> >
> >> I don't think of half a  billion key value pairs as that large a number
> -
> >> nor 20,000 per task - these are
> >> not atypical for hadoop tasks and many users will see these as small
> >> numbers
> >> while you might use cleverness such as a combiner to reduce the output I
> >> wonder if this is needed
> >> What is your cluster size and how fast does the job perform???
> >>
> >
> > I am using combiner to compact the output a little bit before they got
> > written to the disk. My cluster is 48 cores (6 nodes * 8cores/node), my
> > chunk size is 12MB, there are 90 or so map tasks, and it takes about 30
> min
> > to process.  It is very slow I think. Thanks for the attention and
> interest!
> >
> > Boyu
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>

--00151750e3b8ccf9f3048dba3872--