hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xu, Richard " <richard...@citi.com>
Subject RE: What is the best practice of using flushCommit in multithreaded mode
Date Wed, 01 Jun 2011 18:27:15 GMT
Thanks a lot, J-D.

I was using 90.2.

-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
Sent: Wednesday, June 01, 2011 1:17 PM
To: user@hbase.apache.org
Subject: Re: What is the best practice of using flushCommit in multithreaded mode

Inline.

J-D

On Wed, Jun 1, 2011 at 6:34 AM, Xu, Richard <richard.xu@citi.com> wrote:
> Hi folks,
>
> I need to load 1 million queue messages into a hbase table in 30 mins.
>
> As "HBase: The Definitive Guide" suggests, I use Client API, flushCommits().
>
> I launched, say, 20 threads, each thread has its own queue connection and hbase instance,
which read msg from queue and insert it into hbase. At the end of thread life, it will call
the hbaseTable.flushCommits().
>
> It seems working fine, except for:
>
>  1. Each flushCommit action takes quite a long time.

Take a look at http://hbase.apache.org/book/performance.html, it will
take time if you are splitting and moving regions a lot.

>  2. Occasionally, flushCommit cause WrongRegionException.

That should never happen, when it does it's because there's a hole in
your .META. table and that doesn't just come out of nowhere, usually
it's due to a misconfiguration.

Which version are you running?

J-D

Mime
View raw message