hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "HBase Review Board (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2462) Review compaction heuristic and move compaction code out so standalone and independently testable
Date Sat, 23 Oct 2010 07:49:21 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924147#action_12924147
] 

HBase Review Board commented on HBASE-2462:
-------------------------------------------

Message from: "Jonathan Gray" <jgray@apache.org>

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1078/
-----------------------------------------------------------

(Updated 2010-10-23 00:48:28.858137)


Review request for hbase, stack, Nicolas, Karthik Ranganathan, and Kannan Muthukkaruppan.


Changes
-------

Working patch.


Summary
-------

Pulls compaction file selection code into new interface and makes it so it's configurable.
 Currently is globally configurable but should be easy to make it per-family setting.

Also makes the algorithm standalone and testable.

Includes a new compaction algorithm based on a new config param 'compactionForce'.  See javadoc
in compaction classes for explanation.

Big test included for new algorithm.

Also the TestCompact class includes a neat new way for us to compare compaction algorithms.
 You specify a bunch of input paramaters and then it runs a simulation and generates statistics.
 The output looks like:


-----
Ran test
-----
numPuts=1000000
putSizeRange=1.0KB to 10.0KB
numPutsPerGet=10
flushSizeRange=64.0MB to 256.0MB
max=10, threshold=3, force=6, factor=0.5
-----

-----
Final Result
-----
files=82.2MB, 2.9GB, 898.3MB, 1.3GB
memstoreSize=100.8MB
totalSize=5.1GB
totalThroughput=18.2GB
averageFilesPerGet=3.25622


This addresses bug HBASE-2462.
    http://issues.apache.org/jira/browse/HBASE-2462


Diffs (updated)
-----

  trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java 1026565 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java 1026565

  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1026565 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1026565 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java 1026565 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/PriorityCompactionQueue.java 1026565

  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 1026565

  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 1026565 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1026565 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compact/CompactSplitThread.java
PRE-CREATION 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compact/CompactionRequestor.java
PRE-CREATION 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compact/CompactionSelector.java
PRE-CREATION 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compact/CompactionSelectorHBase89.java
PRE-CREATION 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compact/CompactionSelectorWithForce.java
PRE-CREATION 
  trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 1026565 
  trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestPriorityCompactionQueue.java
1026565 
  trunk/src/test/java/org/apache/hadoop/hbase/regionserver/compact/TestCompact.java PRE-CREATION


Diff: http://review.cloudera.org/r/1078/diff


Testing
-------

TestCompact is passing.  Have not run test suite.


Thanks,

Jonathan




> Review compaction heuristic and move compaction code out so standalone and independently
testable
> -------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-2462
>                 URL: https://issues.apache.org/jira/browse/HBASE-2462
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: Jonathan Gray
>            Priority: Critical
>
> Anything that improves our i/o profile makes hbase run smoother.  Over in HBASE-2457,
good work has been done already describing the tension between minimizing compactions versus
minimizing count of store files.  This issue is about following on from what has been done
in 2457 but also, breaking the hard-to-read compaction code out of Store.java out to a standalone
class that can be the easier tested (and easily analyzed for its performance characteristics).
> If possible, in the refactor, we'd allow specification of alternate merge sort implementations.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message