Return-Path: Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: (qmail 97670 invoked from network); 23 Oct 2010 06:36:04 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 23 Oct 2010 06:36:04 -0000 Received: (qmail 81372 invoked by uid 500); 23 Oct 2010 06:36:04 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 81227 invoked by uid 500); 23 Oct 2010 06:36:02 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 81203 invoked by uid 99); 23 Oct 2010 06:36:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 06:36:01 +0000 X-ASF-Spam-Status: No, hits=1.8 required=10.0 tests=FH_HELO_EQ_D_D_D_D,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [184.73.217.71] (HELO ip-10-202-7-187.ec2.internal) (184.73.217.71) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 06:35:56 +0000 Received: from ip-10-202-7-187.ec2.internal (localhost [127.0.0.1]) by ip-10-202-7-187.ec2.internal (Postfix) with ESMTP id 217588A204; Sat, 23 Oct 2010 06:35:36 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Subject: Review Request: Review compaction heuristic and move compaction code out so standalone and independently testable From: "Jonathan Gray" To: "Karthik Ranganathan" , "Kannan Muthukkaruppan" , "Nicolas" , stack@duboce.net Date: Sat, 23 Oct 2010 06:35:36 -0000 Message-ID: <20101023063536.11862.62468@ip-10-202-7-187.ec2.internal> Cc: jiraposter@review.hbase.org, "Jonathan Gray" , dev@hbase.apache.org ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/1078/ ----------------------------------------------------------- Review request for hbase, stack, Nicolas, Karthik Ranganathan, and Kannan M= uthukkaruppan. Summary ------- Pulls compaction file selection code into new interface and makes it so it'= s configurable. Currently is globally configurable but should be easy to m= ake it per-family setting. Also makes the algorithm standalone and testable. Includes a new compaction algorithm based on a new config param 'compaction= Force'. See javadoc in compaction classes for explanation. Big test included for new algorithm. Also the TestCompact class includes a neat new way for us to compare compac= tion algorithms. You specify a bunch of input paramaters and then it runs = a simulation and generates statistics. The output looks like: ----- Ran test ----- numPuts=3D1000000 putSizeRange=3D1.0KB to 10.0KB numPutsPerGet=3D10 flushSizeRange=3D64.0MB to 256.0MB max=3D10, threshold=3D3, force=3D6, factor=3D0.5 ----- ----- Final Result ----- files=3D82.2MB, 2.9GB, 898.3MB, 1.3GB memstoreSize=3D100.8MB totalSize=3D5.1GB totalThroughput=3D18.2GB averageFilesPerGet=3D3.25622 This addresses bug HBASE-2462. http://issues.apache.org/jira/browse/HBASE-2462 Diffs ----- trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java 1026565 = trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionSelect= or.java PRE-CREATION = trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionSelect= orHBase89.java PRE-CREATION = trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionSelect= orWithForce.java PRE-CREATION = trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 10265= 65 = trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompact.java= PRE-CREATION = Diff: http://review.cloudera.org/r/1078/diff Testing ------- TestCompact is passing. Have not run test suite. Thanks, Jonathan