hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From st...@duboce.net
Subject Re: Review Request: Review compaction heuristic and move compaction code out so standalone and independently testable
Date Sat, 23 Oct 2010 06:57:25 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1078/#review1634
-----------------------------------------------------------


This is great.  Don't we already have a standalone compaction issues?  Should kill it w/ this
issue?  Some broad questions below.  Do you find your new selector better than old?


trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionSelector.java
<http://review.cloudera.org/r/1078/#comment5493>

    What is an index range?
    
    Presumption is that we compact adjacent files?  Might want to say that in javadoc?
    



trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionSelectorHBase89.java
<http://review.cloudera.org/r/1078/#comment5494>

    compaction stuff deserves its own subpackage now?



trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
<http://review.cloudera.org/r/1078/#comment5495>

    Would be 'nicer' if selector returned List of StoreFiles rather than indices?



trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
<http://review.cloudera.org/r/1078/#comment5496>

    I suppose compaction is single-threaded.  Later when multi-threaded will have to do better
here.



trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
<http://review.cloudera.org/r/1078/#comment5497>

    Fall back on a default?


- stack


On 2010-10-22 23:39:07, Jonathan Gray wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/1078/
> -----------------------------------------------------------
> 
> (Updated 2010-10-22 23:39:07)
> 
> 
> Review request for hbase, stack, Nicolas, Karthik Ranganathan, and Kannan Muthukkaruppan.
> 
> 
> Summary
> -------
> 
> Pulls compaction file selection code into new interface and makes it so it's configurable.
 Currently is globally configurable but should be easy to make it per-family setting.
> 
> Also makes the algorithm standalone and testable.
> 
> Includes a new compaction algorithm based on a new config param 'compactionForce'.  See
javadoc in compaction classes for explanation.
> 
> Big test included for new algorithm.
> 
> Also the TestCompact class includes a neat new way for us to compare compaction algorithms.
 You specify a bunch of input paramaters and then it runs a simulation and generates statistics.
 The output looks like:
> 
> 
> -----
> Ran test
> -----
> numPuts=1000000
> putSizeRange=1.0KB to 10.0KB
> numPutsPerGet=10
> flushSizeRange=64.0MB to 256.0MB
> max=10, threshold=3, force=6, factor=0.5
> -----
> 
> -----
> Final Result
> -----
> files=82.2MB, 2.9GB, 898.3MB, 1.3GB
> memstoreSize=100.8MB
> totalSize=5.1GB
> totalThroughput=18.2GB
> averageFilesPerGet=3.25622
> 
> 
> This addresses bug HBASE-2462.
>     http://issues.apache.org/jira/browse/HBASE-2462
> 
> 
> Diffs
> -----
> 
>   trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java 1026565 
>   trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionSelector.java PRE-CREATION

>   trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionSelectorHBase89.java
PRE-CREATION 
>   trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionSelectorWithForce.java
PRE-CREATION 
>   trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1026565 
>   trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompact.java PRE-CREATION

> 
> Diff: http://review.cloudera.org/r/1078/diff
> 
> 
> Testing
> -------
> 
> TestCompact is passing.  Have not run test suite.
> 
> 
> Thanks,
> 
> Jonathan
> 
>


Mime
View raw message