asterixdb-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luo Chen (Code Review)" <do-not-re...@asterixdb.incubator.apache.org>
Subject Change in asterixdb[master]: Avoid always merging old components in prefix policy
Date Fri, 09 Jun 2017 04:05:33 GMT
Hello Jenkins,

I'd like you to reexamine a change.  Please visit

    https://asterix-gerrit.ics.uci.edu/1818

to look at the new patch set (#4).

Change subject: Avoid always merging old components in prefix policy
......................................................................

Avoid always merging old components in prefix policy

Current, the prefix policy always looks at the components from
oldest to newest to schedule merge. One negative consequence is that
the oldest (largest) component gets merged over and over again
until it reaches the size limit. This is undesirable since it takes
O(n^2) disk IOs (n is the number of flushed components) to produce a
final component.

This patch is a temporary fix of this behavior, taken from the idea of
HBase compaction policy (https://www.ngdata.com/visualizing-hbase
-flushes-and-compactions/). The basic idea is that it introduces
some size factor (for now it's 1.2) to control the merge behavior.

When the prefix policy finds a sequence of components to merge,
we also check the oldest (largest) component in the sequence should
be smaller than 1.2*the total size of all younger components.
By doing so, we can avoid merging oldest components over and over again,
making the disk IOs O(nlog n).

Change-Id: I464da3fed38cded0aee7b319a35664eae069a2ba
---
M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/PrefixMergePolicy.java
M hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-am-lsm-common-test/pom.xml
A hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-am-lsm-common-test/src/test/java/org/apache/hyracks/storage/am/lsm/common/test/PrefixMergePolicyTest.java
3 files changed, 247 insertions(+), 37 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/18/1818/4
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1818
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I464da3fed38cded0aee7b319a35664eae069a2ba
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen <cluo8@uci.edu>
Gerrit-Reviewer: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Mime
View raw message