hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rajesh balamohan <rbalamoha...@gmail.com>
Subject Re: HBase ImportTSV
Date Thu, 15 Sep 2011 06:01:18 GMT
Thanks a lot for the quick response. It worked like a charm.

On Thu, Sep 15, 2011 at 10:44 AM, Stack <stack@duboce.net> wrote:

> Do you know your keyspace roughly?  Try creating a pre-split table
> with as many regions as you want reducers.
> St.Ack
>
> On Wed, Sep 14, 2011 at 8:25 PM, rajesh balamohan
> <rbalamohan2k@gmail.com> wrote:
> > ImportTSV internally uses HFileOutputFormat.configureIncrementalLoad(job,
> > table);
> >
> > However, for newly created tables there would not be any keys available.
> > Hence, it launches 1 reducer by default.
> >
> > Is there a way to increase the number of reducers for high volume imports
> > like 500+ GB.
> >
> > ~Rajesh.B
> >
> > On Thu, Sep 15, 2011 at 8:51 AM, rajesh balamohan <
> rbalamohan2k@gmail.com>wrote:
> >
> >> Hi All,
> >>
> >> ImportTSV is a great tool for bulk loading the data into HBASE.
> >>
> >> I have close to 500+GB of raw data which I would like to import into a
> >> newly created HTABLE. If I go ahead with ImportTSV, it creates only one
> >> reducer which is a bottleneck in terms of sorting and shuffling.
> >>
> >> Are there any other way, I can increase the number of reducers while
> doing
> >> bulk loads for new table?.
> >>
> >> ~Rajesh.B
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message