hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: HBase ImportTSV
Date Thu, 15 Sep 2011 05:14:22 GMT
Do you know your keyspace roughly?  Try creating a pre-split table
with as many regions as you want reducers.
St.Ack

On Wed, Sep 14, 2011 at 8:25 PM, rajesh balamohan
<rbalamohan2k@gmail.com> wrote:
> ImportTSV internally uses HFileOutputFormat.configureIncrementalLoad(job,
> table);
>
> However, for newly created tables there would not be any keys available.
> Hence, it launches 1 reducer by default.
>
> Is there a way to increase the number of reducers for high volume imports
> like 500+ GB.
>
> ~Rajesh.B
>
> On Thu, Sep 15, 2011 at 8:51 AM, rajesh balamohan <rbalamohan2k@gmail.com>wrote:
>
>> Hi All,
>>
>> ImportTSV is a great tool for bulk loading the data into HBASE.
>>
>> I have close to 500+GB of raw data which I would like to import into a
>> newly created HTABLE. If I go ahead with ImportTSV, it creates only one
>> reducer which is a bottleneck in terms of sorting and shuffling.
>>
>> Are there any other way, I can increase the number of reducers while doing
>> bulk loads for new table?.
>>
>> ~Rajesh.B
>>
>

Mime
View raw message