hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "HBase Review Board (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-3160) Compactions: Use more intelligent priorities for PriorityCompactionQueue
Date Thu, 28 Oct 2010 21:49:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925960#action_12925960
] 

HBase Review Board commented on HBASE-3160:
-------------------------------------------

Message from: "Nicolas" <nspiegelberg@facebook.com>

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/
-----------------------------------------------------------

(Updated 2010-10-28 14:48:04.897593)


Review request for hbase.


Changes
-------

Updates after internal review.
1. Removed PRIORITY_BLOCKING.  It was a little confusing because blocking is <= 0 and not
necessarily the value 0 itself.
2. Cleaned up compaction downgrade conditional.

Questions resolved:
1q. Wondering if we want to go ahead with a split if the compaction priority is PRIORITY_BLOCKING?
1a. Counter-example: storefiles could be very small and compaction will reduce it to one medium
file. It was starved because a major compaction occurred immediately beforehand and took forever
to finish.
2q. Should we add in logic to interrupt a running compaction if it is lower priority (e.g.
Pri >= 5) and we have a PRIORITY_BLOCKING request? We can then re-enqueue the compaction
to run at a less pressured time.
2a. Save this problem for another JIRA.  Since the master has threadpools, we should give
the CompactSplitThread prioritized threadpools and dedicate one thread only to blocking requests.


Summary
-------

Switch to more intelligent priority metric: blockingSize - max(len(s.storefiles) for s in
region.stores) . This will allow us to better prioritize, give us faster responsiveness to
users, and feel more cavalier about issuing new compaction requests.  Note that we also found/fixed
a major compaction downgrade bug while writing this code.


This addresses bug HBASE-3160.
    http://issues.apache.org/jira/browse/HBASE-3160


Diffs (updated)
-----

  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java 1027787

  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionRequestor.java 1027787

  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1027787 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1027787 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java 1027787 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/PriorityCompactionQueue.java 1027787

  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1027787 
  trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestPriorityCompactionQueue.java
1027787 

Diff: http://review.cloudera.org/r/1103/diff


Testing
-------

mvn clean install -Dtest=TestPriorityCompaction
dev cluster tests (on 0.89)


Thanks,

Nicolas




> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
>                 Key: HBASE-3160
>                 URL: https://issues.apache.org/jira/browse/HBASE-3160
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 0.89.20100924, 0.90.0
>            Reporter: Nicolas Spiegelberg
>            Assignee: Nicolas Spiegelberg
>             Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity
on the importance of the various compactions in the queue.  If a StoreFile count exceeds 15
files, only then do we bump via enum change.  We should instead look into more intelligent,
granular priority metrics for choosing the next compaction.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message