Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com
 designates 209.85.217.176 as permitted sender)
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <BLU436-SMTP51744DA75755BD110E13B08F2D0@phx.gbl>
References: <538F139E.6050305@viadeoteam.com>
	<BLU436-SMTP51744DA75755BD110E13B08F2D0@phx.gbl>
Date: Thu, 5 Jun 2014 08:11:25 -0700
Message-ID: 
 <CADcMMgG+DEOF=u=_zb1GDBGSLgD+Zt9Em0LUB7wnicpEVyhMEA@mail.gmail.com>
Subject: Re: HBase export limit bandwith
From: Stack <stack@duboce.net>
To: Hbase-User <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=047d7b3a8c4e95c72604fb18287d

--047d7b3a8c4e95c72604fb18287d
Content-Type: text/plain; charset=UTF-8

>
>
> On Jun 4, 2014, at 7:39 AM, Damien Hardy <dhardy@viadeoteam.com> wrote:
>
> > Hello,
> >
> > We are trying to export HBase table on S3 for backup purpose.
> > By default export tool run a map per region and we want to limit output
> > bandwidth on internet (to amazon s3).
> >
> > We were thinking in adding some reducer to limit the number of writers
> > but this is explicitly hardcoded to 0 in Export class
> > ```
> >    // No reducers. Just write straight to output files.
> >    job.setNumReduceTasks(0);
> > ```
> >


Echoing Michael Segel, why not subclass and set reducers to whatever you
want in your subclass?

But you probably don't want to have reducers anyways.  The output from your
mappers will have to be sorted and fed to the reducers which will put up a
load on your cluster, a loading that could be better deployed moving the
data to S3.

Or limit the number of mappers you have running at any one time via
configuration or in a subclass limit the rate at which they write?

St.Ack

--047d7b3a8c4e95c72604fb18287d--