hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray" <jg...@apache.org>
Subject Re: Review Request: Review compaction heuristic and move compaction code out so standalone and independently testable
Date Sat, 23 Oct 2010 07:48:29 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1078/
-----------------------------------------------------------

(Updated 2010-10-23 00:48:28.858137)


Review request for hbase, stack, Nicolas, Karthik Ranganathan, and Kannan Muthukkaruppan.


Changes
-------

Working patch.


Summary
-------

Pulls compaction file selection code into new interface and makes it so it's configurable.
 Currently is globally configurable but should be easy to make it per-family setting.

Also makes the algorithm standalone and testable.

Includes a new compaction algorithm based on a new config param 'compactionForce'.  See javadoc
in compaction classes for explanation.

Big test included for new algorithm.

Also the TestCompact class includes a neat new way for us to compare compaction algorithms.
 You specify a bunch of input paramaters and then it runs a simulation and generates statistics.
 The output looks like:


-----
Ran test
-----
numPuts=1000000
putSizeRange=1.0KB to 10.0KB
numPutsPerGet=10
flushSizeRange=64.0MB to 256.0MB
max=10, threshold=3, force=6, factor=0.5
-----

-----
Final Result
-----
files=82.2MB, 2.9GB, 898.3MB, 1.3GB
memstoreSize=100.8MB
totalSize=5.1GB
totalThroughput=18.2GB
averageFilesPerGet=3.25622


This addresses bug HBASE-2462.
    http://issues.apache.org/jira/browse/HBASE-2462


Diffs (updated)
-----

  trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java 1026565 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java 1026565

  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1026565 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1026565 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java 1026565 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/PriorityCompactionQueue.java 1026565

  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 1026565

  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 1026565 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1026565 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compact/CompactSplitThread.java
PRE-CREATION 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compact/CompactionRequestor.java
PRE-CREATION 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compact/CompactionSelector.java
PRE-CREATION 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compact/CompactionSelectorHBase89.java
PRE-CREATION 
  trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compact/CompactionSelectorWithForce.java
PRE-CREATION 
  trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 1026565 
  trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestPriorityCompactionQueue.java
1026565 
  trunk/src/test/java/org/apache/hadoop/hbase/regionserver/compact/TestCompact.java PRE-CREATION


Diff: http://review.cloudera.org/r/1078/diff


Testing
-------

TestCompact is passing.  Have not run test suite.


Thanks,

Jonathan


Mime
View raw message