hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Reducing compaction queue more efficiently
Date Thu, 27 Mar 2014 14:52:23 GMT
>From src/main/ruby/hbase/admin.rb :

    def major_compact(table_or_region_name, family = nil)
      if family == nil
        @admin.majorCompact(table_or_region_name)
      else
        # We are major compacting a column family within a region or table.
        @admin.majorCompact(table_or_region_name, family)
      end
    end

In HBaseAdmin:

   * Major compact a table or an individual region.

   * Asynchronous operation.

   *

   * @param tableNameOrRegionName table or region to major compact

   * @throws IOException if a remote or network exception occurs

   * @throws InterruptedException

   */

  public void majorCompact(final String tableNameOrRegionName)


Cheers


On Thu, Mar 27, 2014 at 4:36 AM, Sven Herzky
<sven.herzky@mail.rakuten.com>wrote:

> Hey all,
>
> I've put some data (~ 2.5 TB) into an HBase table on a small cluster (8
> dn/rs + 1 master, max region size 10 GB, having ~ 350 regions) and
> collected something around 1500 entries in my compaction queue (~ 200 per
> regionserver). Unfortunately, HBase now is slow when pushing new data into
> it and it just compacts with 0.02 per second (getting also "WARN
> org.apache.hadoop.ipc.HBaseServer: (operationTooSlow)" in log files).
>
> If I want to trigger the compaction manually, there's nothing happening:
>
> hbase(main):001:0> compact 'images'
> 0 row(s) in 1.3690 seconds
>
> hbase(main):002:0> major_compact 'images'
> 0 row(s) in 2.3670 seconds
>
> I'm currently using cdh4 with HBase 0.94.
>
> Do somebody have a clue how to speed up the process or why the manual
> compaction doesn't work as "expected" (by me)?
>
> Thanks in advance,
> Sven
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message