hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: random access slowly when writing new data to hTable, how to optimize?
Date Thu, 04 Nov 2010 03:19:09 GMT
You can add this to line 3:
        table.setAutoFlush(false);


On Wed, Nov 3, 2010 at 8:01 PM, 刘京磊 <liu.jinglei@goodhope.net> wrote:

> hadoop version:  0.20.2
> hbase version: 0.20.6
>
> 1 master , 7 slaves/regionservers : 4cpus, 34G memory   hbase heap size:
> 15G
> 3 zookeepers: 2cpus, 8G memeory
>
> now, there are 1573 regions(about 1T). It spends 10ms-200ms when random
> accessing if not writing.
>
> We need to put 200G data (about 0.4billion rows) at a time. It maybe spend
> 20+s when random accessing.
>
> code like this:
>
> HTablePool tablePool = new HTablePool(config, 20);
> HTable table = tablePool.getTable("productDB");
> table.setWriteBufferSize(1024*1024*24);
> List<Put> puts = new ArrayList<Put>();
>
> while(true){
>  Put put = new Put(Bytes.toBytes(id));
>  put.add(...);
>  put.setWriteToWAL(false);
>  puts.add(put);
>
>  if(rows % 1000 == 0){
>    table.getWriteBuffer().addAll(puts);
>    table.flushCommits();
>    puts.clear();
>  }
> }
>
>
> Thanks for any help
>
> --
> Derek Liu
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message