hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liu, Ming (Ming)" <ming....@esgyn.cn>
Subject [Help] minor compact is continuously consuming the disk space until run out of space?
Date Sat, 26 Aug 2017 13:48:05 GMT
hi, all,

We have a system with 17 nodes, with a big table about 28T in size. We use native hbase bulkloader
(importtsv) to load data, and it generated a lot of hfiles, on each Region Server there are
about 800 hfiles.  We turned off Major Compact, but the Minor compaction is running due to
so many hfile. 
The problem is, after the initial loading, there are about 80% disk space used, when minor
compaction is going on, we notice the disk space is reducing rapidly until all disk spaces
are used and hbase went down. 

We try to change the hbase.hstore.compactionThreshold to 2000, but the minor compaction is
still triggered. 

The system is CDH 5.7, HBase is 1.2.

Could anyone help to give us some suggestions? We are really stuck. Thanks in advance.

Thanks,
Ming

-----Original Message-----
From: Andrzej [mailto:borucki_andrzej@wp.pl] 
Sent: Friday, August 25, 2017 11:55 PM
To: user@hbase.apache.org
Subject: How to optimal using column family?

I need database to store transactions. Each transaction can have many 
inputs and outputs, usually 1-3 but sometimes about 100.
Previously in sql I have two main tables: first table had one row per 
transaction and store common transaction things; second had one row per 
one input/output.
But now are column families. I need two tables or family input and 
output and many columns like input:col57, output:col26 ? Which is better 
efficient?
Mime
View raw message