hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Outdated data can not be cleaned in time
Date Thu, 09 Jun 2011 05:52:38 GMT
On Wed, Jun 8, 2011 at 9:14 PM, Zhoushuaifeng <zhoushuaifeng@huawei.com> wrote:
> //zhou: I agree regions with more files should take higher priority, but there are other
factors important should be considered. In our test case, we found some regions sent to the
queue by major compact checker hunging in the queue for more than 2 days! Some scanners on
these regions cannot get availably data for a long time and lease expired.
> I think set these regions priority to hbase.hstore.blockingStoreFiles - hbase.hstore.compactionThreshold
-1 as default may be a good way to solve this problem. If regions have less than 3 files,
it's priority is lower than outdated regions, but if it has more than 4 files, it's priority
will be higher. This settings can both solve the outdated problem and will not block flush
and put.

I would not build on hbase.hstore.blockingStoreFiles.  That should be
going away.

What else would you suggest?

> Maybe the compaction algorithm should consider age of compactions too?
>  If a compaction has been hanging the queue a good while, its priority
> gets bumped a level?
> //zhou: This is good, I totally agree. This is another good way to solve the problem.
But by now, I don't know how to make the patch, may be we can dig more on it.

Want to file an issue on this Zhou?


View raw message