Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of kranganathan@fb.com
 designates 67.231.153.30 as permitted sender)
From: Karthik Ranganathan <kranganathan@fb.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>,
        Hbase Dev
	<dev@hbase.apache.org>
Subject: Re: Increase number of reducers for bulk data load to empty HBase
 table
Thread-Topic: Increase number of reducers for bulk data load to empty HBase
 table
Thread-Index: AQHMjXSJ5Lhcw+IgRkiF77vfOdgy5pWD0FAA
Date: Wed, 19 Oct 2011 15:41:50 +0000
Message-ID: <CAC43D7A.EAB4%kranganathan@fb.com>
In-Reply-To: 
 <CADSCZ4R2PiNJz0+Yw5ybt_whtv9Z9n-BBS0HZFcfWb42YJMfFw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.12.0.110505
Content-Type: text/plain; charset="us-ascii"
Content-ID: <7619C34D1D2C2F40BC63E938C4EBB4EF@fb.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hey Matthew,

The only way to increase the number of reducers is to have more regions -
each reducer produces an output per region, so the number of reducers =3D=
=3D
number of regions.

Thanks
Karthik


On 10/18/11 2:00 AM, "Matthew Tovbin" <matthew@tovbin.com> wrote:

>Hello, Guys,
>
>I'm willing to bulk load data from hdfs folders into HBase, for this
>purpose
>I used configureIncrementalLoad method from HFileOutputFormat that
>configures the job, as follows:
>
>org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLo
>ad(job,
>myTable)
>
>The problem is that destination table in HBase is empty, meaning it's only
>hosted by one region server, so the resulted number of reducers is 1,
>which
>makes the job to run almost forever.
>
>How can I increase the number of reducers? Can the number of reducers be
>set
>to more than a number of region servers?
>
>Thanks in advance,
>     Matthew Tovbin.