Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <5D66A842901F8E41815AF6D27A28EC490A8E1ABD73@Mail-Ab02.rmg-ny.com>
References: <5D66A842901F8E41815AF6D27A28EC490A8E0C597D@Mail-Ab02.rmg-ny.com>
	<a6906bde0912220840p7bb8c9c5j96a23dd450174f82@mail.gmail.com>
	<a4a007d40912220917q271902b8gfc6985af22cd9912@mail.gmail.com>
	<5D66A842901F8E41815AF6D27A28EC490A8E0C5ACE@Mail-Ab02.rmg-ny.com>
	<c9118d4c0912221734j528174bbh1bd75210940afa6a@mail.gmail.com>
	<5D66A842901F8E41815AF6D27A28EC490A8E1ABD73@Mail-Ab02.rmg-ny.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 23 Dec 2009 09:31:05 -0800
Message-ID: <45f85f70912230931o30db106cu546529c8f527a8f7@mail.gmail.com>
Subject: Re: io.sort.mb configuration?
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e90c3d1e867e047b68b123

--001636e90c3d1e867e047b68b123
Content-Type: text/plain; charset=ISO-8859-1

Hi Mark,

For what it's wroth, you're unlikely to see a big difference in performance
unless you cut down the number of spills from >1 to 1, or >io.sort.factor to
<io.sort.factor. The difference between 3 spills and 5 spills is not huge,
in my experience, since you're still writing the same amount of data to
disk.

If your output data should fit well within io.sort.mb, but you're seeing a
few map spills per task instead, that's when tuning io.sort.record.percent
yields a big improvement.

I believe it says so somewhere on that JIRA, but the magic number for
io.sort.record.percent is 16/(16 + average record size in bytes). You can
calculate the average record size by simply looking at the job counters and
dividing map output bytes by map output records.

-Todd

On Wed, Dec 23, 2009 at 9:22 AM, Mark Vigeant
<mark.vigeant@riskmetrics.com>wrote:

> Oh Cool Jeff, this is really helpful!
>
> I actually have been dealing with relatively *small* records (of about 500
> MB) and running very simple programs akin to wordcount so I will play around
> with io.sort.record.percent and see what results I get.
>
> Thanks!
>
> -----Original Message-----
> From: Jeff Hammerbacher [mailto:hammer@cloudera.com]
> Sent: Tuesday, December 22, 2009 8:35 PM
> To: common-user@hadoop.apache.org
> Subject: Re: io.sort.mb configuration?
>
> Hey Mark,
>
> While you're grokking this aspect of MapReduce's configuration, you may
> want
> to check out https://issues.apache.org/jira/browse/MAPREDUCE-64, which is
> on
> its way into trunk right now. Chris Douglas from Yahoo! has posted a very
> nice explanation of how buffers are managed during the shuffle and which
> parameters affect the behavior.
>
> Regards,
> Jeff
>
> On Tue, Dec 22, 2009 at 12:30 PM, Mark Vigeant <
> mark.vigeant@riskmetrics.com
> > wrote:
>
> > Thank you for the responses guys!
> >
> > First, to Patrick, I didn't set it in the code, though I will try it
> > because that's a really good idea to set it there, so I shall play around
> > with that.
> >
> > Long: I should have clarified, I am using 0.20.1, and so this is a bit
> > different. I set the parameter in mapred-site.xml and for some reason
> it's
> > just not getting implemented. Thank you anyways, though!
> >
> > -Mark
> >
> > -----Original Message-----
> > From: Long Van Nguyen Dinh [mailto:muntron@gmail.com]
> > Sent: Tuesday, December 22, 2009 12:17 PM
> > To: common-user@hadoop.apache.org
> > Subject: Re: io.sort.mb configuration?
> >
> > Hadoop has a default file (hadoop-default.xml - version 19) for all
> > configuration, don't change the values in that file (they won't be
> > affected), copy the parameter to the file hadoop-site.xml where you
> > set up the cluster and set the value you want there.
> >
> > Long Van
> >
> > On Tue, Dec 22, 2009 at 11:40 AM, Patrick Angeles
> > <patrickangeles@gmail.com> wrote:
> > > You can also set that param per-job. Maybe you called some code that
> did
> > > that behind the scenes?
> > >
> > > On Tue, Dec 22, 2009 at 11:10 AM, Mark Vigeant <
> > mark.vigeant@riskmetrics.com
> > >> wrote:
> > >
> > >> Hey Everyone-
> > >>
> > >> I've been playing around with Hadoop and Hbase for a while and I
> noticed
> > >> that when running a program to upload data into an HTable I saw the
> > output:
> > >>
> > >> INFO mapred.MapTask: io.sort.mb = 100
> > >>
> > >> Which is the default value, but in the mapred configuration on all
> > machines
> > >> in my cluster I set this value to 250. Could it be that my program is
> > not
> > >> accessing the configuration properly? Is that too large a value? Or is
> > it
> > >> most likely just a foolish syntax error on my part?
> > >>
> > >> Thank you very much, all input is appreciated.
> > >>
> > >> Mark Vigeant
> > >> RiskMetrics Group, Inc.
> > >>
> > >>
> > >> This email message and any attachments are for the sole use of the
> > intended
> > >> recipients and may contain proprietary and/or confidential information
> > which
> > >> may be privileged or otherwise protected from disclosure. Any
> > unauthorized
> > >> review, use, disclosure or distribution is prohibited. If you are not
> an
> > >> intended recipient, please contact the sender by reply email and
> destroy
> > the
> > >> original message and any copies of the message as well as any
> > attachments to
> > >> the original message.
> > >>
> > >
> >
> > This email message and any attachments are for the sole use of the
> intended
> > recipients and may contain proprietary and/or confidential information
> which
> > may be privileged or otherwise protected from disclosure. Any
> unauthorized
> > review, use, disclosure or distribution is prohibited. If you are not an
> > intended recipient, please contact the sender by reply email and destroy
> the
> > original message and any copies of the message as well as any attachments
> to
> > the original message.
> >
>
> This email message and any attachments are for the sole use of the intended
> recipients and may contain proprietary and/or confidential information which
> may be privileged or otherwise protected from disclosure. Any unauthorized
> review, use, disclosure or distribution is prohibited. If you are not an
> intended recipient, please contact the sender by reply email and destroy the
> original message and any copies of the message as well as any attachments to
> the original message.
>

--001636e90c3d1e867e047b68b123--