accumulo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [1/2] accumulo git commit: Add a paragraph on the smaller keys for index blocks improvement
Date Thu, 23 Jun 2016 15:12:41 GMT
Repository: accumulo
Updated Branches:
  refs/heads/asf-site 45d81dfc4 -> 604500f88
  refs/heads/gh-pages 6538290f3 -> f30cc51b1

Add a paragraph on the smaller keys for index blocks improvement


Branch: refs/heads/gh-pages
Commit: f30cc51b11c0c2c0289f19579ca78a98753e547f
Parents: 6538290
Author: Josh Elser <>
Authored: Thu Jun 23 11:11:05 2016 -0400
Committer: Josh Elser <>
Committed: Thu Jun 23 11:11:05 2016 -0400

 release_notes/ | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/release_notes/ b/release_notes/
index eebdc70..dccb092 100644
--- a/release_notes/
+++ b/release_notes/
@@ -31,10 +31,14 @@ There was a bug ([ACCUMULO-4148][ACCUMULO-4148]) where multiple put calls
with i
 An improvement was introduced to allow a max age before WAL files would be automatically
rolled. Without a max age, they could stay open for writing indefinitely, blocking the Hadoop
DataNode decomissioning process. For more information, see [ACCUMULO-4004][ACCUMULO-4004].
-### Remove copy of cached RFile index blocks
+### Remove unnecessary copy of cached RFile index blocks
 Accumulo maintains an cache for file blocks in-memory as a performance optimization. This
can be done safely because Accumulo RFiles are immutable, thus their blocks are also immutable.
There are two types of these blocks: index and data blocks. Index blocks refer to the b-tree
style index inside of each Accumulo RFile, while data blocks contain the sorted Key-Value
pairs. In previous versions, when Accumulo extracted an Index block from the in-memory cache,
it would copy the data. [ACCUMULO-4164][ACCUMULO-4164] removes this unnecessary copy as the
contents are immutable and can be passed by reference. Ensuring that the Index blocks are
not copied when accessed from the cache is a big performance gain at the file-access level.
+### Analyze Key-length to avoid choosing large Keys for RFile Index blocks
+Accumulo's RFile index blocks are made up of a Key which exists in the file and points to
that specific location in the corresponding RFile data block. Thus, the size of the RFile
index blocks is largely dominated by the size of the Keys which are used by the index. [ACCUMULO-4314][ACCUMULO-4314]
is an improvement which uses statistics on the length of the Keys in the Rfile to avoid choosing
Keys for the index whose length is greater than three standard deviations for the RFile. By
choosing smaller Keys for the index, Accumulo can access the RFile index faster and keep more
Index blocks cached in memory. Initial tests showed that with this change, the RFile index
size was nearly cut in half.
 ### Minor performance improvements.
 Tablet servers would previously always hsync at the start of a minor compaction, causing
delays in the write pipeline. These additional syncs were determined to provide no additional
durability guarantees and have been removed. See [ACCUMULO-4112][ACCUMULO-4112] for additional
@@ -83,3 +87,4 @@ HDFS High-Availability instances, forcing NameNode failover.

View raw message