hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhoushuaifeng <zhoushuaif...@huawei.com>
Subject RE: Outdated data can not be cleaned in time
Date Thu, 09 Jun 2011 07:05:15 GMT

-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Thursday, June 09, 2011 1:53 PM
To: dev@hbase.apache.org
Cc: Yanlijun; Chenjian
Subject: Re: Outdated data can not be cleaned in time

On Wed, Jun 8, 2011 at 9:14 PM, Zhoushuaifeng <zhoushuaifeng@huawei.com> wrote:
> //zhou: I agree regions with more files should take higher priority, but there are other
factors important should be considered. In our test case, we found some regions sent to the
queue by major compact checker hunging in the queue for more than 2 days! Some scanners on
these regions cannot get availably data for a long time and lease expired.
> I think set these regions priority to hbase.hstore.blockingStoreFiles - hbase.hstore.compactionThreshold
-1 as default may be a good way to solve this problem. If regions have less than 3 files,
it's priority is lower than outdated regions, but if it has more than 4 files, it's priority
will be higher. This settings can both solve the outdated problem and will not block flush
and put.
>

I would not build on hbase.hstore.blockingStoreFiles.  That should be
going away.

What else would you suggest?

//zhou: other way is when major compact checker send regions to the compact queue, consider
how long the oldest timestamp passed the TTL. The older get the higher priority.
And what do you mean by 'That should be going away' ? I mean change like this in the compaction
checker:
+      majorCompactPriority = this.instance.conf.getInt("hbase.regionserver.compactionChecker.majorCompactPriority",
+    		  this.instance.conf.getInt("hbase.hstore.blockingStoreFiles", 7)
+    		  - this.instance.conf.getInt("hbase.hstore.compactionThreshold", 3)
+    		  -1);

if (s.isMajorCompaction()) {
+                // Queue a compaction. Will recognize if major is needed.
+                this.instance.compactSplitThread.requestCompaction(r, s,
+                    getName() + " requests major compaction", this.majorCompactPriority);
+              }


> Maybe the compaction algorithm should consider age of compactions too?
>  If a compaction has been hanging the queue a good while, its priority
> gets bumped a level?
> //zhou: This is good, I totally agree. This is another good way to solve the problem.
But by now, I don't know how to make the patch, may be we can dig more on it.
>

Want to file an issue on this Zhou?
//ok, I will file an issue, and provide the possible solution and patch.

St.Ack

Mime
View raw message