Return-Path: X-Original-To: apmail-hbase-commits-archive@www.apache.org Delivered-To: apmail-hbase-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2138F7FA6 for ; Tue, 11 Oct 2011 02:20:45 +0000 (UTC) Received: (qmail 83202 invoked by uid 500); 11 Oct 2011 02:20:45 -0000 Delivered-To: apmail-hbase-commits-archive@hbase.apache.org Received: (qmail 83168 invoked by uid 500); 11 Oct 2011 02:20:44 -0000 Mailing-List: contact commits-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list commits@hbase.apache.org Received: (qmail 83161 invoked by uid 99); 11 Oct 2011 02:20:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Oct 2011 02:20:44 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Oct 2011 02:20:41 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 012642388BEF for ; Tue, 11 Oct 2011 02:20:21 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1181559 - in /hbase/branches/0.89/src: main/java/org/apache/hadoop/hbase/io/hfile/ main/java/org/apache/hadoop/hbase/mapreduce/ main/java/org/apache/hadoop/hbase/regionserver/ main/java/org/apache/hadoop/hbase/util/ main/java/org/apache/ha... Date: Tue, 11 Oct 2011 02:20:19 -0000 To: commits@hbase.apache.org From: nspiegelberg@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20111011022021.012642388BEF@eris.apache.org> Author: nspiegelberg Date: Tue Oct 11 02:20:18 2011 New Revision: 1181559 URL: http://svn.apache.org/viewvc?rev=1181559&view=rev Log: Unit test changes, HFile reader/writer API call changes, and utility functions for HFile format version 2 Summary: This is the sixth diff for the HFile version 2 / compound Bloom filters / multi-level block indexes project. It bundles the following types of changes: - API calls to create HFile readers/writes now have to go through the appropriate factory methods. - Unit test changes to enable testing both HFile format version 1 and 2. The set of tests to be made parameterized was selected in an ad-hoc way during development. Certain unit tests had to be converted to JUnit 4, but still extend HBaseTestCase, to minimize code changes. Now that the default HFile format for this feature has been flipped to version 2, we might be able to avoid a lot of these unit test changes by running two rounds of test with the default HFile format version specified through e.g. a system property. Also, testHeapSizeForBlockIndex was moved from TestHFile to block index's own unit test, TestHFileBlockIndex. - New utility functions required for other parts of HFile v2 functionality (e.g. changes to Bytes and TestBytes). - A fix for a typo in Hash.java javadoc. Test Plan: Unit tests. Load testing with HBase test. Will push to dark launch along with other HFile format v2 changes. Reviewed By: nspiegelberg Reviewers: kannan, liyintang, nspiegelberg, kranganathan, gqchen, aaiyer CC: hbase@lists, , mbautin, nspiegelberg Revert Plan: HFile format v2 changes are backwards-compatible but not reverse-compatible. Differential Revision: 252110 Added: hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/Bytes.java hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/Hash.java hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/manual/HBaseTest.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/regionserver/CreateRandomStoreFile.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java (original) +++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java Tue Oct 11 02:20:18 2011 @@ -55,22 +55,22 @@ public class CachedBlock implements Heap }; private final String blockName; - private final ByteBuffer buf; + private final HeapSize buf; private volatile long accessTime; private long size; private BlockPriority priority; - public CachedBlock(String blockName, ByteBuffer buf, long accessTime) { + public CachedBlock(String blockName, HeapSize buf, long accessTime) { this(blockName, buf, accessTime, false); } - public CachedBlock(String blockName, ByteBuffer buf, long accessTime, + public CachedBlock(String blockName, HeapSize buf, long accessTime, boolean inMemory) { this.blockName = blockName; this.buf = buf; this.accessTime = accessTime; - this.size = ClassSize.align(blockName.length()) + - ClassSize.align(buf.capacity()) + PER_BLOCK_OVERHEAD; + this.size = ClassSize.align(blockName.length()) + + ClassSize.align(buf.heapSize()) + PER_BLOCK_OVERHEAD; if(inMemory) { this.priority = BlockPriority.MEMORY; } else { @@ -97,7 +97,7 @@ public class CachedBlock implements Heap return this.accessTime < that.accessTime ? 1 : -1; } - public ByteBuffer getBuffer() { + public HeapSize getBuffer() { return this.buf; } Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java (original) +++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java Tue Oct 11 02:20:18 2011 @@ -76,7 +76,7 @@ public class HFileOutputFormat extends F // Get the path of the temporary output file final Path outputPath = FileOutputFormat.getOutputPath(context); final Path outputdir = new FileOutputCommitter(outputPath, context).getWorkPath(); - Configuration conf = context.getConfiguration(); + final Configuration conf = context.getConfiguration(); final FileSystem fs = outputdir.getFileSystem(conf); // These configs. are from hbase-*.xml final long maxsize = conf.getLong("hbase.hregion.max.filesize", @@ -131,7 +131,7 @@ public class HFileOutputFormat extends F // create a new HLog writer, if necessary if (wl == null || wl.writer == null) { - wl = getNewWriter(family); + wl = getNewWriter(family, conf); } // we now have the proper HLog writer. full steam ahead @@ -161,12 +161,13 @@ public class HFileOutputFormat extends F * @return A WriterLength, containing a new HFile.Writer. * @throws IOException */ - private WriterLength getNewWriter(byte[] family) throws IOException { + private WriterLength getNewWriter(byte[] family, Configuration conf) + throws IOException { WriterLength wl = new WriterLength(); Path familydir = new Path(outputdir, Bytes.toString(family)); String compression = compressionMap.get(family); compression = compression == null ? defaultCompression : compression; - wl.writer = new HFile.Writer(fs, + wl.writer = HFile.getWriterFactory(conf).createWriter(fs, StoreFile.getUniqueFile(fs, familydir), blocksize, bytesPerChecksum, compression, KeyValue.KEY_COMPARATOR); this.writers.put(family, wl); Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java (original) +++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java Tue Oct 11 02:20:18 2011 @@ -184,7 +184,7 @@ public class LoadIncrementalHFiles exten throws IOException { final Path hfilePath = item.hfilePath; final FileSystem fs = hfilePath.getFileSystem(getConf()); - HFile.Reader hfr = new HFile.Reader(fs, hfilePath, null, false, false); + HFile.Reader hfr = HFile.createReader(fs, hfilePath, null, false, false); final byte[] first, last; try { hfr.loadFileInfo(); @@ -276,7 +276,7 @@ public class LoadIncrementalHFiles exten halfWriter = new StoreFile.Writer( fs, outFile, blocksize, compression, conf, KeyValue.COMPARATOR, - bloomFilterType, 0, false); + bloomFilterType, 0); HFileScanner scanner = halfReader.getScanner(false, false, false); scanner.seekTo(); do { Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java (original) +++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java Tue Oct 11 02:20:18 2011 @@ -270,6 +270,13 @@ public class HRegion implements HeapSize oldVal.addAndGet(amount); } + static long getNumericMetric(String key) { + AtomicLong m = numericMetrics.get(key); + if (m == null) + return 0; + return m.get(); + } + public static void incrTimeVaryingMetric(String key, long amount) { Pair oldVal = timeVaryingMetrics.get(key); if (oldVal == null) { Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java (original) +++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java Tue Oct 11 02:20:18 2011 @@ -24,10 +24,8 @@ import java.io.InterruptedIOException; import java.util.ArrayList; import java.util.Collection; import java.util.Collections; -import java.util.HashSet; import java.util.List; import java.util.NavigableSet; -import java.util.Set; import java.util.SortedSet; import java.util.concurrent.CopyOnWriteArraySet; import java.util.concurrent.locks.ReentrantReadWriteLock; @@ -60,7 +58,6 @@ import com.google.common.collect.Collect import com.google.common.collect.ImmutableList; import com.google.common.collect.Iterables; import com.google.common.collect.Lists; -import com.google.common.collect.Sets; /** * A Store holds a column family in a Region. Its a memstore and a set of zero @@ -327,7 +324,7 @@ public class Store implements HeapSize { try { LOG.info("Validating hfile at " + srcPath + " for inclusion in " + "store " + this + " region " + this.region); - reader = new HFile.Reader(srcPath.getFileSystem(conf), + reader = HFile.createReader(srcPath.getFileSystem(conf), srcPath, null, false, false); reader.loadFileInfo(); @@ -519,8 +516,7 @@ public class Store implements HeapSize { throws IOException { return StoreFile.createWriter(this.fs, region.getTmpDir(), this.blocksize, this.compression, this.comparator, this.conf, - this.family.getBloomFilterType(), maxKeyCount, - conf.getBoolean("hbase.rs.cacheblocksonwrite", false)); + this.family.getBloomFilterType(), maxKeyCount); } /* @@ -571,7 +567,7 @@ public class Store implements HeapSize { */ void deleteChangedReaderObserver(ChangedReadersObserver o) { if (!this.changedReaderObservers.remove(o)) { - LOG.warn("Not in set" + o); + LOG.warn("Not in set " + o); } } @@ -1030,7 +1026,8 @@ public class Store implements HeapSize { LOG.info("Compacting: " + file + "; keyCount = " + keyCount + "; Bloom Type = " + r.getBloomFilterType().toString() + - "; Size = " + StringUtils.humanReadableInt(r.length()) ); + "; Size = " + StringUtils.humanReadableInt(r.length()) + + "; HFile v" + r.getHFileVersion()); } } LOG.info("Estimated total keyCount for output of compaction = " + maxKeyCount); @@ -1520,6 +1517,37 @@ public class Store implements HeapSize { } /** + * Returns the total size of all index blocks in the data block indexes, + * including the root level, intermediate levels, and the leaf level for + * multi-level indexes, or just the root level for single-level indexes. + * + * @return the total size of block indexes in the store + */ + long getTotalStaticIndexSize() { + long size = 0; + for (StoreFile s : storefiles) { + size += s.getReader().getUncompressedDataIndexSize(); + } + return size; + } + + /** + * Returns the total byte size of all Bloom filter bit arrays. For compound + * Bloom filters even the Bloom blocks currently not loaded into the block + * cache are counted. + * + * @return the total size of all Bloom filters in the store + */ + long getTotalStaticBloomSize() { + long size = 0; + for (StoreFile s : storefiles) { + StoreFile.Reader r = s.getReader(); + size += r.getTotalBloomSize(); + } + return size; + } + + /** * @return The priority that this store should have in the compaction queue */ public int getCompactPriority() { Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/Bytes.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/Bytes.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/Bytes.java (original) +++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/Bytes.java Tue Oct 11 02:20:18 2011 @@ -298,17 +298,22 @@ public class Bytes { * @see #toStringBinary(byte[], int, int) */ public static String toStringBinary(final byte [] b) { + if (b == null) + return "null"; return toStringBinary(b, 0, b.length); } /** - * The same as {@link #toStringBinary(byte[])}, but returns a string "null" - * if given a null argument. + * Converts the given byte buffer, from its array offset to its limit, to + * a string. The position and the mark are ignored. + * + * @param buf a byte buffer + * @return a string representation of the buffer's binary contents */ - public static String toStringBinarySafe(final byte [] b) { - if (b == null) + public static String toStringBinary(ByteBuffer buf) { + if (buf == null) return "null"; - return toStringBinary(b, 0, b.length); + return toStringBinary(buf.array(), buf.arrayOffset(), buf.limit()); } /** @@ -1150,12 +1155,18 @@ public class Bytes { /** * Binary search for keys in indexes. + * * @param arr array of byte arrays to search for * @param key the key you want to find * @param offset the offset in the key you want to find * @param length the length of the key * @param comparator a comparator to compare. - * @return index of key + * @return zero-based index of the key, if the key is present in the array. + * Otherwise, a value -(i + 1) such that the key is between arr[i - + * 1] and arr[i] non-inclusively, where i is in [0, i], if we define + * arr[-1] = -Inf and arr[N] = Inf for an N-element array. The above + * means that this function can return 2N + 1 different values + * ranging from -(N + 1) to N - 1. */ public static int binarySearch(byte [][]arr, byte []key, int offset, int length, RawComparator comparator) { @@ -1265,4 +1276,34 @@ public class Bytes { return value; } + /** + * Writes a string as a fixed-size field, padded with zeros. + */ + public static void writeStringFixedSize(final DataOutput out, String s, + int size) throws IOException { + byte[] b = toBytes(s); + if (b.length > size) { + throw new IOException("Trying to write " + b.length + " bytes (" + + toStringBinary(b) + ") into a field of length " + size); + } + + out.writeBytes(s); + for (int i = 0; i < size - s.length(); ++i) + out.writeByte(0); + } + + /** + * Reads a fixed-size field and interprets it as a string padded with zeros. + */ + public static String readStringFixedSize(final DataInput in, int size) + throws IOException { + byte[] b = new byte[size]; + in.readFully(b); + int n = b.length; + while (n > 0 && b[n - 1] == 0) + --n; + + return toString(b, 0, n); + } + } Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java (original) +++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java Tue Oct 11 02:20:18 2011 @@ -21,6 +21,7 @@ package org.apache.hadoop.hbase.util; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.io.hfile.HFile; import org.apache.hadoop.hdfs.DistributedFileSystem; @@ -61,16 +62,18 @@ public class CompressionTest { public static void main(String[] args) { if (args.length != 2) usage(); + Configuration conf = HBaseConfiguration.create(); try { DistributedFileSystem dfs = openConnection(args[0]); dfs.delete(path, false); - HFile.Writer writer = new HFile.Writer(dfs, path, - HFile.DEFAULT_BLOCKSIZE, HFile.DEFAULT_BYTES_PER_CHECKSUM, args[1], null); + HFile.Writer writer = HFile.getWriterFactory(conf).createWriter(dfs, + path, HFile.DEFAULT_BLOCKSIZE, HFile.DEFAULT_BYTES_PER_CHECKSUM, + args[1], null); writer.append(Bytes.toBytes("testkey"), Bytes.toBytes("testval")); writer.appendFileInfo(Bytes.toBytes("infokey"), Bytes.toBytes("infoval")); writer.close(); - HFile.Reader reader = new HFile.Reader(dfs, path, null, false, false); + HFile.Reader reader = HFile.createReader(dfs, path, null, false, false); reader.loadFileInfo(); byte[] key = reader.getFirstKey(); boolean rc = Bytes.toString(key).equals("testkey"); Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/Hash.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/Hash.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/Hash.java (original) +++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/util/Hash.java Tue Oct 11 02:20:18 2011 @@ -122,8 +122,8 @@ public abstract class Hash { } /** - * Calculate a hash using bytes from 0 to length, and - * the provided seed value + * Calculate a hash using bytes from offset to offset + + * length, and the provided seed value. * @param bytes input bytes * @param offset the offset into the array to start consideration * @param length length of the valid bytes after offset to consider Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java (original) +++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java Tue Oct 11 02:20:18 2011 @@ -472,8 +472,8 @@ public class RecoverableZooKeeper { return path; } LOG.error("Node " + path + " already exists with " + - Bytes.toStringBinarySafe(currentData) + ", could not write " + - Bytes.toStringBinarySafe(data)); + Bytes.toStringBinary(currentData) + ", could not write " + + Bytes.toStringBinary(data)); throw e; } LOG.error("Node " + path + " already exists and this is not a " + Modified: hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java (original) +++ hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java Tue Oct 11 02:20:18 2011 @@ -31,7 +31,6 @@ import java.util.Arrays; import java.util.List; import java.util.Map; import java.util.NavigableSet; -import java.util.Set; import java.util.UUID; import org.apache.commons.logging.Log; @@ -50,6 +49,7 @@ import org.apache.hadoop.hbase.client.Pu import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.ResultScanner; import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.io.hfile.Compression; import org.apache.hadoop.hbase.master.HMaster; import org.apache.hadoop.hbase.regionserver.HRegionServer; import org.apache.hadoop.hbase.regionserver.InternalScanner; @@ -101,6 +101,19 @@ public class HBaseTestingUtility { */ public static final String DEFAULT_TEST_DIRECTORY = "target/build/data"; + /** Compression algorithms to use in parameterized JUnit 4 tests */ + public static final List COMPRESSION_ALGORITHMS_PARAMETERIZED = + Arrays.asList(new Object[][] { + { Compression.Algorithm.NONE }, + { Compression.Algorithm.GZ } + }); + + /** Compression algorithms to use in testing */ + public static final Compression.Algorithm[] COMPRESSION_ALGORITHMS = + new Compression.Algorithm[] { + Compression.Algorithm.NONE, Compression.Algorithm.GZ + }; + public HBaseTestingUtility() { this(HBaseConfiguration.create()); } @@ -117,6 +130,18 @@ public class HBaseTestingUtility { } /** + * Makes sure the test directory is set up so that {@link #getTestDir()} + * returns a valid directory. Useful in unit tests that do not run a + * mini-cluster. + */ + public void initTestDir() { + if (System.getProperty(TEST_DIRECTORY_KEY) == null) { + clusterTestBuildDir = setupClusterTestBuildDir(); + System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath()); + } + } + + /** * @return Where to write test data on local filesystem; usually * {@link #DEFAULT_TEST_DIRECTORY} * @see #setupClusterTestBuildDir() Modified: hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java (original) +++ hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java Tue Oct 11 02:20:18 2011 @@ -188,9 +188,9 @@ public class HFilePerformanceEvaluation @Override void setUp() throws Exception { - writer = new HFile.Writer(this.fs, this.mf, RFILE_BLOCKSIZE, - HFile.DEFAULT_BYTES_PER_CHECKSUM, (Compression.Algorithm) null, null, - null); + writer = HFile.getWriterFactory(conf).createWriter(this.fs, this.mf, + RFILE_BLOCKSIZE, HFile.DEFAULT_BYTES_PER_CHECKSUM, + (Compression.Algorithm) null, null); } @Override @@ -226,7 +226,7 @@ public class HFilePerformanceEvaluation @Override void setUp() throws Exception { - reader = new HFile.Reader(this.fs, this.mf, null, false, false); + reader = HFile.createReader(this.fs, this.mf, null, false, false); this.reader.loadFileInfo(); } Modified: hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java (original) +++ hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java Tue Oct 11 02:20:18 2011 @@ -20,6 +20,7 @@ package org.apache.hadoop.hbase.io; +import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.HBaseTestingUtility; @@ -60,9 +61,10 @@ public class TestHalfStoreFileReader { String root_dir = HBaseTestingUtility.getTestDir("TestHalfStoreFile").toString(); Path p = new Path(root_dir, "test"); - FileSystem fs = FileSystem.get(test_util.getConfiguration()); + Configuration conf = test_util.getConfiguration(); + FileSystem fs = FileSystem.get(conf); - HFile.Writer w = new HFile.Writer(fs, p, 1024, + HFile.Writer w = HFile.getWriterFactory(conf).createWriter(fs, p, 1024, HFile.DEFAULT_BYTES_PER_CHECKSUM, "none", KeyValue.KEY_COMPARATOR); // write some things. @@ -72,7 +74,7 @@ public class TestHalfStoreFileReader { } w.close(); - HFile.Reader r = new HFile.Reader(fs, p, null, false, false); + HFile.Reader r = HFile.createReader(fs, p, null, false, false); r.loadFileInfo(); byte [] midkey = r.midkey(); KeyValue midKV = KeyValue.createKeyValueFromKey(midkey); Modified: hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java (original) +++ hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java Tue Oct 11 02:20:18 2011 @@ -68,9 +68,9 @@ public class RandomSeek { long start = System.currentTimeMillis(); SimpleBlockCache cache = new SimpleBlockCache(); //LruBlockCache cache = new LruBlockCache(); - Reader reader = new HFile.Reader(lfs, path, cache, false, false); + Reader reader = HFile.createReader(lfs, path, cache, false, false); reader.loadFileInfo(); - System.out.println(reader.trailer); + System.out.println(reader.getTrailer()); long end = System.currentTimeMillis(); System.out.println("Index read time: " + (end - start)); Added: hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java?rev=1181559&view=auto ============================================================================== --- hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java (added) +++ hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java Tue Oct 11 02:20:18 2011 @@ -0,0 +1,210 @@ +/* + * Copyright 2011 The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.io.hfile; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.EnumMap; +import java.util.List; +import java.util.Random; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.io.hfile.BlockCache; +import org.apache.hadoop.hbase.io.hfile.Compression; +import org.apache.hadoop.hbase.io.hfile.HFile; +import org.apache.hadoop.hbase.io.hfile.HFileBlock; +import org.apache.hadoop.hbase.io.hfile.HFileBlockIndex; +import org.apache.hadoop.hbase.io.hfile.HFileReaderV2; +import org.apache.hadoop.hbase.io.hfile.HFileScanner; +import org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2; +import org.apache.hadoop.hbase.regionserver.CreateRandomStoreFile; +import org.apache.hadoop.hbase.regionserver.StoreFile; +import org.apache.hadoop.hbase.util.BloomFilterFactory; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; +import org.junit.runners.Parameterized.Parameters; +import static org.junit.Assert.*; + +/** + * Tests {@link HFile} cache-on-write functionality for the following block + * types: data blocks, non-root index blocks, and Bloom filter blocks. + */ +@RunWith(Parameterized.class) +public class TestCacheOnWrite { + + private static final Log LOG = LogFactory.getLog(TestCacheOnWrite.class); + + private static final HBaseTestingUtility TEST_UTIL = + new HBaseTestingUtility(); + private Configuration conf; + private FileSystem fs; + private Random rand = new Random(12983177L); + private Path storeFilePath; + private Compression.Algorithm compress; + private CacheOnWriteType cowType; + private BlockCache blockCache; + private String testName; + + private static final int DATA_BLOCK_SIZE = 2048; + private static final int NUM_KV = 25000; + private static final int INDEX_BLOCK_SIZE = 512; + private static final int BLOOM_BLOCK_SIZE = 4096; + + private static enum CacheOnWriteType { + DATA_BLOCKS(BlockType.DATA, HFile.CACHE_BLOCKS_ON_WRITE_KEY), + BLOOM_BLOCKS(BlockType.BLOOM_CHUNK, + BloomFilterFactory.IO_STOREFILE_BLOOM_CACHE_ON_WRITE), + INDEX_BLOCKS(BlockType.LEAF_INDEX, + HFileBlockIndex.CACHE_INDEX_BLOCKS_ON_WRITE_KEY); + + private final String confKey; + private final BlockType inlineBlockType; + + private CacheOnWriteType(BlockType inlineBlockType, String confKey) { + this.inlineBlockType = inlineBlockType; + this.confKey = confKey; + } + + public boolean shouldBeCached(BlockType blockType) { + return blockType == inlineBlockType + || blockType == BlockType.INTERMEDIATE_INDEX + && inlineBlockType == BlockType.LEAF_INDEX; + } + + public void modifyConf(Configuration conf) { + for (CacheOnWriteType cowType : CacheOnWriteType.values()) + conf.setBoolean(cowType.confKey, cowType == this); + } + + } + + public TestCacheOnWrite(CacheOnWriteType cowType, + Compression.Algorithm compress) { + this.cowType = cowType; + this.compress = compress; + testName = "[cacheOnWrite=" + cowType + ", compress=" + compress + "]"; + } + + @Parameters + public static Collection getParameters() { + List cowTypes = new ArrayList(); + for (CacheOnWriteType cowType : CacheOnWriteType.values()) + for (Compression.Algorithm compress : + HBaseTestingUtility.COMPRESSION_ALGORITHMS) { + cowTypes.add(new Object[] { cowType, compress }); + } + return cowTypes; + } + + @Before + public void setUp() throws IOException { + conf = TEST_UTIL.getConfiguration(); + conf.setInt(HFile.FORMAT_VERSION_KEY, HFile.MAX_FORMAT_VERSION); + conf.setInt(HFileBlockIndex.MAX_CHUNK_SIZE_KEY, INDEX_BLOCK_SIZE); + conf.setInt(BloomFilterFactory.IO_STOREFILE_BLOOM_BLOCK_SIZE, + BLOOM_BLOCK_SIZE); + cowType.modifyConf(conf); + fs = FileSystem.get(conf); + blockCache = StoreFile.getBlockCache(conf); + } + + @After + public void tearDown() { + blockCache.evictBlocksByPrefix(""); + } + + @Test + public void testCacheOnWrite() throws IOException { + writeStoreFile(); + readStoreFile(); + } + + private void readStoreFile() throws IOException { + HFileReaderV2 reader = (HFileReaderV2) HFile.createReader(fs, + storeFilePath, null, false, false); + LOG.info("HFile information: " + reader); + HFileScanner scanner = reader.getScanner(false, false); + assertTrue(testName, scanner.seekTo()); + + long offset = 0; + HFileBlock prevBlock = null; + EnumMap blockCountByType = + new EnumMap(BlockType.class); + + while (offset < reader.getTrailer().getLoadOnOpenDataOffset()) { + HFileBlock block = reader.readBlockData(offset, prevBlock == null ? -1 + : prevBlock.getNextBlockOnDiskSizeWithHeader(), -1, false); + String blockCacheKey = HFile.getBlockCacheKey(reader.getName(), offset); + boolean isCached = blockCache.getBlock(blockCacheKey) != null; + boolean shouldBeCached = cowType.shouldBeCached(block.getBlockType()); + assertEquals(testName + " " + block, shouldBeCached, isCached); + prevBlock = block; + offset += block.getOnDiskSizeWithHeader(); + BlockType bt = block.getBlockType(); + Integer count = blockCountByType.get(bt); + blockCountByType.put(bt, (count == null ? 0 : count) + 1); + } + + LOG.info("Block count by type: " + blockCountByType); + assertEquals( + "{DATA=1367, LEAF_INDEX=172, BLOOM_CHUNK=9, INTERMEDIATE_INDEX=24}", + blockCountByType.toString()); + + reader.close(); + } + + public void writeStoreFile() throws IOException { + Path storeFileParentDir = new Path(HBaseTestingUtility.getTestDir(), + "test_cache_on_write"); + StoreFile.Writer sfw = StoreFile.createWriter(fs, storeFileParentDir, + DATA_BLOCK_SIZE, compress, KeyValue.COMPARATOR, conf, + StoreFile.BloomType.ROWCOL, NUM_KV); + + final int rowLen = 32; + for (int i = 0; i < NUM_KV; ++i) { + byte[] k = TestHFileWriterV2.randomOrderedKey(rand, i); + byte[] v = TestHFileWriterV2.randomValue(rand); + int cfLen = rand.nextInt(k.length - rowLen + 1); + KeyValue kv = new KeyValue( + k, 0, rowLen, + k, rowLen, cfLen, + k, rowLen + cfLen, k.length - rowLen - cfLen, + rand.nextLong(), + CreateRandomStoreFile.generateKeyType(rand), + v, 0, v.length); + sfw.append(kv); + } + + sfw.close(); + storeFilePath = sfw.getPath(); + } + +} Modified: hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java (original) +++ hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java Tue Oct 11 02:20:18 2011 @@ -20,6 +20,9 @@ package org.apache.hadoop.hbase.io.hfile; import java.nio.ByteBuffer; + +import org.apache.hadoop.hbase.io.HeapSize; + import junit.framework.TestCase; public class TestCachedBlockQueue extends TestCase { @@ -124,9 +127,13 @@ public class TestCachedBlockQueue extend private static class CachedBlock extends org.apache.hadoop.hbase.io.hfile.CachedBlock { - public CachedBlock(long heapSize, String name, long accessTime) { + public CachedBlock(final long heapSize, String name, long accessTime) { super(name, - ByteBuffer.allocate((int)(heapSize - CachedBlock.PER_BLOCK_OVERHEAD)), + new HeapSize(){ + @Override + public long heapSize() { + return ((int)(heapSize - CachedBlock.PER_BLOCK_OVERHEAD)); + }}, accessTime,false); } } Modified: hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java (original) +++ hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java Tue Oct 11 02:20:18 2011 @@ -33,11 +33,9 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.HBaseTestCase; import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.KeyValue.KeyComparator; -import org.apache.hadoop.hbase.io.hfile.HFile.BlockIndex; import org.apache.hadoop.hbase.io.hfile.HFile.Reader; import org.apache.hadoop.hbase.io.hfile.HFile.Writer; import org.apache.hadoop.hbase.util.Bytes; -import org.apache.hadoop.hbase.util.ClassSize; import org.apache.hadoop.io.Writable; /** @@ -63,9 +61,9 @@ public class TestHFile extends HBaseTest */ public void testEmptyHFile() throws IOException { Path f = new Path(ROOT_DIR, getName()); - Writer w = new Writer(this.fs, f); + Writer w = HFile.getWriterFactory(conf).createWriter(this.fs, f); w.close(); - Reader r = new Reader(fs, f, null, false, false); + Reader r = HFile.createReader(fs, f, null, false, false); r.loadFileInfo(); assertNull(r.getFirstKey()); assertNull(r.getLastKey()); @@ -134,13 +132,13 @@ public class TestHFile extends HBaseTest void basicWithSomeCodec(String codec) throws IOException { Path ncTFile = new Path(ROOT_DIR, "basic.hfile"); FSDataOutputStream fout = createFSOutput(ncTFile); - Writer writer = new Writer(fout, minBlockSize, - Compression.getCompressionAlgorithmByName(codec), null); + Writer writer = HFile.getWriterFactory(conf).createWriter(fout, + minBlockSize, Compression.getCompressionAlgorithmByName(codec), null); LOG.info(writer); writeRecords(writer); fout.close(); FSDataInputStream fin = fs.open(ncTFile); - Reader reader = new Reader(ncTFile, fs.open(ncTFile), + Reader reader = HFile.createReader(ncTFile, fs.open(ncTFile), fs.getFileStatus(ncTFile).getLen(), null, false, false); // Load up the index. reader.loadFileInfo(); @@ -209,13 +207,14 @@ public class TestHFile extends HBaseTest private void metablocks(final String compress) throws Exception { Path mFile = new Path(ROOT_DIR, "meta.hfile"); FSDataOutputStream fout = createFSOutput(mFile); - Writer writer = new Writer(fout, minBlockSize, - Compression.getCompressionAlgorithmByName(compress), null); + Writer writer = HFile.getWriterFactory(conf).createWriter(fout, + minBlockSize, Compression.getCompressionAlgorithmByName(compress), + null); someTestingWithMetaBlock(writer); writer.close(); fout.close(); FSDataInputStream fin = fs.open(mFile); - Reader reader = new Reader(mFile, fs.open(mFile), + Reader reader = HFile.createReader(mFile, fs.open(mFile), this.fs.getFileStatus(mFile).getLen(), null, false, false); reader.loadFileInfo(); // No data -- this should return false. @@ -233,33 +232,35 @@ public class TestHFile extends HBaseTest } public void testNullMetaBlocks() throws Exception { - Path mFile = new Path(ROOT_DIR, "nometa.hfile"); - FSDataOutputStream fout = createFSOutput(mFile); - Writer writer = new Writer(fout, minBlockSize, - Compression.Algorithm.NONE, null); - writer.append("foo".getBytes(), "value".getBytes()); - writer.close(); - fout.close(); - Reader reader = new Reader(fs, mFile, null, false, false); - reader.loadFileInfo(); - assertNull(reader.getMetaBlock("non-existant", false)); + for (Compression.Algorithm compressAlgo : + HBaseTestingUtility.COMPRESSION_ALGORITHMS) { + Path mFile = new Path(ROOT_DIR, "nometa_" + compressAlgo + ".hfile"); + FSDataOutputStream fout = createFSOutput(mFile); + Writer writer = HFile.getWriterFactory(conf).createWriter(fout, + minBlockSize, compressAlgo, null); + writer.append("foo".getBytes(), "value".getBytes()); + writer.close(); + fout.close(); + Reader reader = HFile.createReader(fs, mFile, null, false, false); + reader.loadFileInfo(); + assertNull(reader.getMetaBlock("non-existant", false)); + } } /** * Make sure the orginals for our compression libs doesn't change on us. */ public void testCompressionOrdinance() { - //assertTrue(Compression.Algorithm.LZO.ordinal() == 0); + assertTrue(Compression.Algorithm.LZO.ordinal() == 0); assertTrue(Compression.Algorithm.GZ.ordinal() == 1); assertTrue(Compression.Algorithm.NONE.ordinal() == 2); } - public void testComparator() throws IOException { Path mFile = new Path(ROOT_DIR, "meta.tfile"); FSDataOutputStream fout = createFSOutput(mFile); - Writer writer = new Writer(fout, minBlockSize, (Compression.Algorithm) null, - new KeyComparator() { + Writer writer = HFile.getWriterFactory(conf).createWriter(fout, + minBlockSize, (Compression.Algorithm) null, new KeyComparator() { @Override public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) { @@ -277,27 +278,4 @@ public class TestHFile extends HBaseTest writer.close(); } - /** - * Checks if the HeapSize calculator is within reason - */ - @SuppressWarnings("unchecked") - public void testHeapSizeForBlockIndex() throws IOException{ - Class cl = null; - long expected = 0L; - long actual = 0L; - - cl = BlockIndex.class; - expected = ClassSize.estimateBase(cl, false); - BlockIndex bi = new BlockIndex(Bytes.BYTES_RAWCOMPARATOR); - actual = bi.heapSize(); - //Since the arrays in BlockIndex(byte [][] blockKeys, long [] blockOffsets, - //int [] blockDataSizes) are all null they are not going to show up in the - //HeapSize calculation, so need to remove those array costs from ecpected. - expected -= ClassSize.align(3 * ClassSize.ARRAY); - if(expected != actual) { - ClassSize.estimateBase(cl, true); - assertEquals(expected, actual); - } - } - } Modified: hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java?rev=1181559&r1=1181558&r2=1181559&view=diff ============================================================================== --- hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java (original) +++ hbase/branches/0.89/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java Tue Oct 11 02:20:18 2011 @@ -157,8 +157,8 @@ public class TestHFilePerformance extend if ("HFile".equals(fileType)){ System.out.println("HFile write method: "); - HFile.Writer writer = - new HFile.Writer(fout, minBlockSize, codecName, null); + HFile.Writer writer = HFile.getWriterFactory(conf).createWriter(fout, + minBlockSize, codecName, null); // Writing value in one shot. for (long l=0 ; l