Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: unknown (nike.apache.org: error in processing during lookup of
 dave@urbanairship.com)
MIME-Version: 1.0
In-Reply-To: <BLU0-SMTP1326441B6BE969D1B0E03F48F160@phx.gbl>
References: 
 <CANH3+J2Act8HG39Ub_66ot1RO034NxqFg0HAXx2Vk5k2OmNwmA@mail.gmail.com>
	<CAFebPXBY5VG46V0vK2C7+1gYMmn2s41TrWFuqsJNvPTJLgD99g@mail.gmail.com>
	<CANH3+J1vMM7W+zLuUvcijm_OP597DVjzvC9j_7qRGPtrVAGp-A@mail.gmail.com>
	<BLU0-SMTP333FD40A9A615AAE40D781E8F160@phx.gbl>
	<CANH3+J1Lv_As5PTz+fgyQFaEpXkeJEgCjhYL7uNJ+6VbbuBPiQ@mail.gmail.com>
	<BLU0-SMTP32016BCB2814BE474EAD17E8F160@phx.gbl>
	<CAN_xCOemLVuoCHJ-WZ-RY0+vmTimh3+qADHX=YJY3-ys5U3Gow@mail.gmail.com>
	<BLU0-SMTP1326441B6BE969D1B0E03F48F160@phx.gbl>
Date: Thu, 10 May 2012 11:41:17 -0700
Message-ID: 
 <CAN_xCOcEY0hBwySTzqrNiBW3nV=oaTvGKwCpsFd7pvQidBk60A@mail.gmail.com>
Subject: Re: Occasional regionserver crashes following socket errors writing
 to HDFS
From: Dave Revell <dave@urbanairship.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=e89a8f83a5a118fbd004bfb2f7a1

--e89a8f83a5a118fbd004bfb2f7a1
Content-Type: text/plain; charset=ISO-8859-1

Some examples of when you'd want a reducer:
http://static.usenix.org/event/osdi04/tech/full_papers/dean/dean.pdf

On Thu, May 10, 2012 at 11:30 AM, Michael Segel
<michael_segel@hotmail.com>wrote:

> Dave, do you really want to go there?
>
> OP has a couple of issues and he was going down a rabbit hole.
> (You can choose if that's a reference to 'the Matrix, Jefferson Starship,
> Alice in Wonderland... or all of the above)
>
> So to put him on the correct path, I recommended the following, not in any
> order...
>
> 1) Increase his region size for this table only.
> 2) Look to decreasing the number of regions managed by a RS (which is why
> you increase region size)
> 3) Up the dfs.balance.bandwidthPerSec. (How often does HBase move regions
> and how exactly do they move regions ?)
> 4) Look at implementing MSLABS and GC tuning. This cuts down on the
> overhead.
> 5) Refactoring his job....
>
> Oops.
> Ok I didn't put that in the list.
> But that was the last thing I wrote as a separate statement.
> Clearly you didn't take my advice and think about the problem....
>
> To prove a point.... you wrote:
> 'Many mapreduce algorithms require a reduce phase (e.g. sorting)'
>
> Ok. So tell me why you would want to sort your input in to HBase and if
> that's really a good thing?
> Oops!... :-)
>
>
>
>
>
>
> On May 10, 2012, at 12:31 PM, Dave Revell wrote:
> > This "you don't need a reducer" conversation is distracting from the real
> > problem and is false.
> >
> > Many mapreduce algorithms require a reduce phase (e.g. sorting). The fact
> > that the output is written to HBase or somewhere else is irrelevant.
> >
> > -Dave
> >
> > On Thu, May 10, 2012 at 6:26 AM, Michael Segel <
> michael_segel@hotmail.com>wrote:
> > [SNIP]
>
>

--e89a8f83a5a118fbd004bfb2f7a1--