Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A83D7200D0F for ; Fri, 15 Sep 2017 08:10:56 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id A664F1609CD; Fri, 15 Sep 2017 06:10:56 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 518741609CF for ; Fri, 15 Sep 2017 08:10:54 +0200 (CEST) Received: (qmail 44747 invoked by uid 500); 15 Sep 2017 06:10:53 -0000 Mailing-List: contact commits-help@asterixdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.apache.org Delivered-To: mailing list commits@asterixdb.apache.org Received: (qmail 44640 invoked by uid 99); 15 Sep 2017 06:10:53 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Sep 2017 06:10:53 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id E2025F5561; Fri, 15 Sep 2017 06:10:52 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: tillw@apache.org To: commits@asterixdb.apache.org Date: Fri, 15 Sep 2017 06:10:54 -0000 Message-Id: In-Reply-To: <4c59cbcf91b042759bb3b5913437f30e@git.apache.org> References: <4c59cbcf91b042759bb3b5913437f30e@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [3/3] asterixdb git commit: [STO][IDX] Eliminated excess antimatter in LSMBTree archived-at: Fri, 15 Sep 2017 06:10:56 -0000 [STO][IDX] Eliminated excess antimatter in LSMBTree - user model changes: no - storage format changes: no - interface changes: no Details: A combination of some LSM operations (e.g. insert+delete) inserts a record into the memory component of LSMBTree and then deletes it right after leaving an antimatter entry. When no flush happens between two operations this "tombstone" entry does not have any purpose and could be eliminated during the flush without changing search semantics. The fix introduces a new bit in record header which tracks if an entry was inserted and then updated in-place. For secondary indexes this will happen only when the record is changed from regular to antimatter. The patch does not introduce changes in storage format because the bit exists only for memory components. In addition the patch refactored *TupleWriters, *TupleWriterFactories, *TupleReferences, *Frames, *FrameFactories to return index-specific types. Change-Id: I12a67eff8431b52d1f9051b793a5a64b15c009e9 Reviewed-on: https://asterix-gerrit.ics.uci.edu/1538 Sonar-Qube: Jenkins Reviewed-by: Till Westmann Tested-by: Jenkins Contrib: Jenkins Integration-Tests: Jenkins Project: http://git-wip-us.apache.org/repos/asf/asterixdb/repo Commit: http://git-wip-us.apache.org/repos/asf/asterixdb/commit/e66346a3 Tree: http://git-wip-us.apache.org/repos/asf/asterixdb/tree/e66346a3 Diff: http://git-wip-us.apache.org/repos/asf/asterixdb/diff/e66346a3 Branch: refs/heads/master Commit: e66346a34776c2386f782904ca3f5ca3fe8d2f1e Parents: f52bc58 Author: Ildar Absalyamov Authored: Thu Sep 14 17:23:57 2017 -0700 Committer: Ildar Absalyamov Committed: Thu Sep 14 21:43:37 2017 -0700 ---------------------------------------------------------------------- .../storage/am/btree/api/IBTreeLeafFrame.java | 2 +- .../compressors/FieldPrefixCompressor.java | 23 +- .../am/btree/dataflow/BTreeResource.java | 2 +- .../frames/BTreeFieldPrefixNSMLeafFrame.java | 25 +- .../BTreeFieldPrefixNSMLeafFrameFactory.java | 10 +- .../frames/BTreeNSMInteriorFrameFactory.java | 8 +- .../am/btree/frames/BTreeNSMLeafFrame.java | 5 - .../btree/frames/BTreeNSMLeafFrameFactory.java | 12 +- .../hyracks/storage/am/btree/impls/BTree.java | 10 +- .../impls/BTreeFieldPrefixTupleReference.java | 146 +++++++ .../btree/impls/FieldPrefixTupleReference.java | 136 ------- .../tuples/BTreeTypeAwareTupleReference.java | 62 +++ .../btree/tuples/BTreeTypeAwareTupleWriter.java | 43 ++ .../BTreeTypeAwareTupleWriterFactory.java | 38 ++ .../storage/am/btree/util/BTreeUtils.java | 21 +- .../common/api/IBTreeIndexTupleReference.java | 35 ++ .../am/common/api/ITreeIndexFrameFactory.java | 1 + .../am/common/api/ITreeIndexTupleWriter.java | 10 + .../api/ITreeIndexTupleWriterFactory.java | 3 +- .../am/common/frames/TreeIndexNSMFrame.java | 8 +- .../am/common/tuples/SimpleTupleReference.java | 3 +- .../am/common/tuples/SimpleTupleWriter.java | 5 +- .../common/tuples/TypeAwareTupleReference.java | 3 +- .../am/common/tuples/TypeAwareTupleWriter.java | 8 +- .../tuples/TypeAwareTupleWriterFactory.java | 6 +- .../am/common/util/BitOperationUtils.java | 38 ++ .../btree/dataflow/LSMBTreeLocalResource.java | 4 +- .../storage/am/lsm/btree/impls/LSMBTree.java | 10 +- .../btree/tuples/LSMBTreeCopyTupleWriter.java | 4 +- .../tuples/LSMBTreeCopyTupleWriterFactory.java | 16 +- .../btree/tuples/LSMBTreeTupleReference.java | 25 +- .../lsm/btree/tuples/LSMBTreeTupleWriter.java | 46 ++- .../tuples/LSMBTreeTupleWriterFactory.java | 15 +- .../am/lsm/btree/utils/LSMBTreeUtil.java | 30 +- .../lsm/common/api/ILSMTreeTupleReference.java | 3 + .../invertedindex/impls/LSMInvertedIndex.java | 2 +- .../inmemory/InMemoryInvertedIndex.java | 2 +- .../ondisk/OnDiskInvertedIndex.java | 2 +- .../invertedindex/util/InvertedIndexUtils.java | 4 +- .../am/lsm/rtree/impls/AbstractLSMRTree.java | 3 +- .../storage/am/lsm/rtree/impls/LSMRTree.java | 3 +- .../impls/LSMRTreeWithAntiMatterTuples.java | 3 +- .../tuples/LSMRTreeCopyTupleWriterFactory.java | 10 +- .../rtree/tuples/LSMRTreeTupleReference.java | 15 +- .../LSMRTreeTupleReferenceForPointMBR.java | 36 +- .../lsm/rtree/tuples/LSMRTreeTupleWriter.java | 18 +- .../tuples/LSMRTreeTupleWriterFactory.java | 9 +- .../LSMRTreeTupleWriterFactoryForPointMBR.java | 8 +- .../tuples/LSMRTreeTupleWriterForPointMBR.java | 12 +- .../tuples/LSMTypeAwareTupleWriterFactory.java | 48 --- .../am/lsm/rtree/utils/LSMRTreeUtils.java | 49 +-- .../am/rtree/frames/RTreeFrameFactory.java | 51 +++ .../frames/RTreeNSMInteriorFrameFactory.java | 26 +- .../am/rtree/frames/RTreeNSMLeafFrame.java | 10 +- .../rtree/frames/RTreeNSMLeafFrameFactory.java | 26 +- .../tuples/RTreeTypeAwareTupleReference.java | 33 ++ .../RTreeTypeAwareTupleWriterFactory.java | 10 +- .../storage/am/rtree/util/RTreeUtils.java | 4 + .../storage/am/btree/BTreeExamplesTest.java | 2 +- .../BTreeModificationOperationCallbackTest.java | 2 +- .../storage/am/btree/BTreeSearchCursorTest.java | 5 +- .../btree/BTreeSearchOperationCallbackTest.java | 2 +- .../storage/am/btree/BTreeStatsTest.java | 4 +- .../storage/am/btree/BTreeUpdateSearchTest.java | 4 +- .../storage/am/btree/FieldPrefixNSMTest.java | 5 +- .../btree/multithread/BTreeMultiThreadTest.java | 2 +- .../storage/am/btree/util/BTreeTestContext.java | 2 +- .../am/lsm/btree/LSMBTreeBulkLoadTest.java | 2 +- .../am/lsm/btree/LSMBTreeDeleteTest.java | 2 +- .../am/lsm/btree/LSMBTreeExamplesTest.java | 2 +- .../am/lsm/btree/LSMBTreeFileManagerTest.java | 2 +- .../am/lsm/btree/LSMBTreeFilterMergeTest.java | 2 +- .../am/lsm/btree/LSMBTreeInsertTest.java | 2 +- .../am/lsm/btree/LSMBTreeLifecycleTest.java | 2 +- .../storage/am/lsm/btree/LSMBTreeMergeTest.java | 2 +- ...MBTreeModificationOperationCallbackTest.java | 2 +- .../am/lsm/btree/LSMBTreeMultiBulkLoadTest.java | 2 +- .../btree/LSMBTreeScanDiskComponentsTest.java | 2 +- .../LSMBTreeSearchOperationCallbackTest.java | 2 +- ...TreeUpdateInPlaceScanDiskComponentsTest.java | 408 +++++++++++++++++++ .../am/lsm/btree/LSMBTreeUpdateInPlaceTest.java | 179 ++++++++ .../am/lsm/btree/LSMBTreeUpdateTest.java | 2 +- .../multithread/LSMBTreeMultiThreadTest.java | 2 +- .../storage/am/lsm/btree/perf/BTreeRunner.java | 2 +- .../btree/perf/ConcurrentSkipListRunner.java | 4 +- .../am/lsm/btree/perf/InMemoryBTreeRunner.java | 4 +- .../am/lsm/btree/perf/InMemorySortRunner.java | 4 +- .../am/lsm/btree/perf/LSMTreeRunner.java | 3 +- .../am/lsm/btree/tuples/LSMBTreeTuplesTest.java | 10 +- .../am/lsm/btree/util/LSMBTreeTestContext.java | 12 +- 90 files changed, 1363 insertions(+), 519 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/api/IBTreeLeafFrame.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/api/IBTreeLeafFrame.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/api/IBTreeLeafFrame.java index ea1bd5f..2a3c1eb 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/api/IBTreeLeafFrame.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/api/IBTreeLeafFrame.java @@ -30,7 +30,7 @@ import org.apache.hyracks.storage.common.buffercache.IExtraPageBlockHelper; public interface IBTreeLeafFrame extends IBTreeFrame { public int findTupleIndex(ITupleReference searchKey, ITreeIndexTupleReference pageTuple, MultiComparator cmp, - FindTupleMode ftm, FindTupleNoExactMatchPolicy ftp) throws HyracksDataException; + FindTupleMode ftm, FindTupleNoExactMatchPolicy ftp) throws HyracksDataException; public int findUpdateTupleIndex(ITupleReference tuple) throws HyracksDataException; http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/compressors/FieldPrefixCompressor.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/compressors/FieldPrefixCompressor.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/compressors/FieldPrefixCompressor.java index ac14b94..499a01a 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/compressors/FieldPrefixCompressor.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/compressors/FieldPrefixCompressor.java @@ -29,11 +29,11 @@ import org.apache.hyracks.api.dataflow.value.ITypeTraits; import org.apache.hyracks.api.exceptions.HyracksDataException; import org.apache.hyracks.storage.am.btree.api.IPrefixSlotManager; import org.apache.hyracks.storage.am.btree.frames.BTreeFieldPrefixNSMLeafFrame; +import org.apache.hyracks.storage.am.btree.impls.BTreeFieldPrefixTupleReference; import org.apache.hyracks.storage.am.btree.impls.FieldPrefixSlotManager; -import org.apache.hyracks.storage.am.btree.impls.FieldPrefixTupleReference; +import org.apache.hyracks.storage.am.btree.tuples.BTreeTypeAwareTupleWriter; import org.apache.hyracks.storage.am.common.api.ITreeIndexFrame; import org.apache.hyracks.storage.am.common.api.ITreeIndexFrameCompressor; -import org.apache.hyracks.storage.am.common.tuples.TypeAwareTupleWriter; import org.apache.hyracks.storage.common.MultiComparator; public class FieldPrefixCompressor implements ITreeIndexFrameCompressor { @@ -163,8 +163,9 @@ public class FieldPrefixCompressor implements ITreeIndexFrameCompressor { int prefixTupleIndex = 0; uncompressedTupleCount = 0; - TypeAwareTupleWriter tupleWriter = new TypeAwareTupleWriter(typeTraits); - FieldPrefixTupleReference tupleToWrite = new FieldPrefixTupleReference(tupleWriter.createTupleReference()); + BTreeTypeAwareTupleWriter tupleWriter = new BTreeTypeAwareTupleWriter(typeTraits, false); + BTreeFieldPrefixTupleReference tupleToWrite = + new BTreeFieldPrefixTupleReference(tupleWriter.createTupleReference()); tupleToWrite.setFieldCount(fieldCount); while (tupleIndex < tupleCount) { @@ -178,11 +179,12 @@ public class FieldPrefixCompressor implements ITreeIndexFrameCompressor { int segmentStart = keyPartitions.get(kpIndex).firstTupleIndex; int tuplesInSegment = 1; - FieldPrefixTupleReference prevTuple = new FieldPrefixTupleReference( - tupleWriter.createTupleReference()); + BTreeFieldPrefixTupleReference prevTuple = + new BTreeFieldPrefixTupleReference(tupleWriter.createTupleReference()); prevTuple.setFieldCount(fieldCount); - FieldPrefixTupleReference tuple = new FieldPrefixTupleReference(tupleWriter.createTupleReference()); + BTreeFieldPrefixTupleReference tuple = + new BTreeFieldPrefixTupleReference(tupleWriter.createTupleReference()); tuple.setFieldCount(fieldCount); for (int i = tupleIndex + 1; i <= keyPartitions.get(kpIndex).lastTupleIndex; i++) { @@ -337,12 +339,13 @@ public class FieldPrefixCompressor implements ITreeIndexFrameCompressor { KeyPartition kp = new KeyPartition(maxCmps); keyPartitions.add(kp); - TypeAwareTupleWriter tupleWriter = new TypeAwareTupleWriter(typeTraits); + BTreeTypeAwareTupleWriter tupleWriter = new BTreeTypeAwareTupleWriter(typeTraits, false); - FieldPrefixTupleReference prevTuple = new FieldPrefixTupleReference(tupleWriter.createTupleReference()); + BTreeFieldPrefixTupleReference prevTuple = + new BTreeFieldPrefixTupleReference(tupleWriter.createTupleReference()); prevTuple.setFieldCount(fieldCount); - FieldPrefixTupleReference tuple = new FieldPrefixTupleReference(tupleWriter.createTupleReference()); + BTreeFieldPrefixTupleReference tuple = new BTreeFieldPrefixTupleReference(tupleWriter.createTupleReference()); tuple.setFieldCount(fieldCount); kp.firstTupleIndex = 0; http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/dataflow/BTreeResource.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/dataflow/BTreeResource.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/dataflow/BTreeResource.java index 24d2a37..b9ad1b1 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/dataflow/BTreeResource.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/dataflow/BTreeResource.java @@ -56,7 +56,7 @@ public class BTreeResource implements IResource { IIOManager ioManager = ctx.getIoManager(); FileReference resourceRef = ioManager.resolve(path); return BTreeUtils.createBTree(bufferCache, typeTraits, comparatorFactories, BTreeLeafFrameType.REGULAR_NSM, - resourceRef, pageManagerFactory.createPageManager(bufferCache)); + resourceRef, pageManagerFactory.createPageManager(bufferCache), false); } @Override http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeFieldPrefixNSMLeafFrame.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeFieldPrefixNSMLeafFrame.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeFieldPrefixNSMLeafFrame.java index 7d74a50..9624158 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeFieldPrefixNSMLeafFrame.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeFieldPrefixNSMLeafFrame.java @@ -30,28 +30,25 @@ import org.apache.hyracks.dataflow.common.data.accessors.ITupleReference; import org.apache.hyracks.storage.am.btree.api.IBTreeLeafFrame; import org.apache.hyracks.storage.am.btree.api.IPrefixSlotManager; import org.apache.hyracks.storage.am.btree.compressors.FieldPrefixCompressor; +import org.apache.hyracks.storage.am.btree.impls.BTreeFieldPrefixTupleReference; import org.apache.hyracks.storage.am.btree.impls.BTreeOpContext.PageValidationInfo; import org.apache.hyracks.storage.am.btree.impls.FieldPrefixPrefixTupleReference; import org.apache.hyracks.storage.am.btree.impls.FieldPrefixSlotManager; -import org.apache.hyracks.storage.am.btree.impls.FieldPrefixTupleReference; +import org.apache.hyracks.storage.am.btree.tuples.BTreeTypeAwareTupleWriter; +import org.apache.hyracks.storage.am.common.api.IBTreeIndexTupleReference; import org.apache.hyracks.storage.am.common.api.ISplitKey; import org.apache.hyracks.storage.am.common.api.ITreeIndexFrame; import org.apache.hyracks.storage.am.common.api.ITreeIndexFrameCompressor; import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleReference; -import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleWriter; import org.apache.hyracks.storage.am.common.frames.FrameOpSpaceStatus; import org.apache.hyracks.storage.am.common.ophelpers.FindTupleMode; import org.apache.hyracks.storage.am.common.ophelpers.FindTupleNoExactMatchPolicy; import org.apache.hyracks.storage.am.common.ophelpers.SlotOffTupleOff; -import org.apache.hyracks.storage.am.common.tuples.TypeAwareTupleWriter; import org.apache.hyracks.storage.common.MultiComparator; import org.apache.hyracks.storage.common.buffercache.IBufferCache; import org.apache.hyracks.storage.common.buffercache.ICachedPage; import org.apache.hyracks.storage.common.buffercache.IExtraPageBlockHelper; -/** - * WARNING: only works when tupleWriter is an instance of TypeAwareTupleWriter - */ public class BTreeFieldPrefixNSMLeafFrame implements IBTreeLeafFrame { protected static final int PAGE_LSN_OFFSET = ITreeIndexFrame.Constants.RESERVED_HEADER_SIZE; @@ -63,21 +60,21 @@ public class BTreeFieldPrefixNSMLeafFrame implements IBTreeLeafFrame { private final IPrefixSlotManager slotManager; private final ITreeIndexFrameCompressor compressor; - private final FieldPrefixTupleReference frameTuple; + private final BTreeFieldPrefixTupleReference frameTuple; private final FieldPrefixPrefixTupleReference framePrefixTuple; - private final ITreeIndexTupleWriter tupleWriter; + private final BTreeTypeAwareTupleWriter tupleWriter; private MultiComparator cmp; protected ICachedPage page = null; protected ByteBuffer buf = null; - public BTreeFieldPrefixNSMLeafFrame(ITreeIndexTupleWriter tupleWriter) { + public BTreeFieldPrefixNSMLeafFrame(BTreeTypeAwareTupleWriter tupleWriter) { this.tupleWriter = tupleWriter; - this.frameTuple = new FieldPrefixTupleReference(tupleWriter.createTupleReference()); + this.frameTuple = new BTreeFieldPrefixTupleReference(tupleWriter.createTupleReference()); this.slotManager = new FieldPrefixSlotManager(); - ITypeTraits[] typeTraits = ((TypeAwareTupleWriter) tupleWriter).getTypeTraits(); + ITypeTraits[] typeTraits = tupleWriter.getTypeTraits(); this.framePrefixTuple = new FieldPrefixPrefixTupleReference(typeTraits); this.compressor = new FieldPrefixCompressor(typeTraits, 0.001f, 2); } @@ -748,13 +745,13 @@ public class BTreeFieldPrefixNSMLeafFrame implements IBTreeLeafFrame { } @Override - public ITreeIndexTupleWriter getTupleWriter() { + public BTreeTypeAwareTupleWriter getTupleWriter() { return tupleWriter; } @Override - public ITreeIndexTupleReference createTupleReference() { - return new FieldPrefixTupleReference(tupleWriter.createTupleReference()); + public IBTreeIndexTupleReference createTupleReference() { + return new BTreeFieldPrefixTupleReference(tupleWriter.createTupleReference()); } @Override http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeFieldPrefixNSMLeafFrameFactory.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeFieldPrefixNSMLeafFrameFactory.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeFieldPrefixNSMLeafFrameFactory.java index f7b270c..47a42b7 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeFieldPrefixNSMLeafFrameFactory.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeFieldPrefixNSMLeafFrameFactory.java @@ -20,16 +20,15 @@ package org.apache.hyracks.storage.am.btree.frames; import org.apache.hyracks.storage.am.btree.api.IBTreeLeafFrame; +import org.apache.hyracks.storage.am.btree.tuples.BTreeTypeAwareTupleWriterFactory; import org.apache.hyracks.storage.am.common.api.ITreeIndexFrameFactory; -import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleWriterFactory; public class BTreeFieldPrefixNSMLeafFrameFactory implements ITreeIndexFrameFactory { - private static final long serialVersionUID = 1L; - private final ITreeIndexTupleWriterFactory tupleWriterFactory; + protected final BTreeTypeAwareTupleWriterFactory tupleWriterFactory; - public BTreeFieldPrefixNSMLeafFrameFactory(ITreeIndexTupleWriterFactory tupleWriterFactory) { + public BTreeFieldPrefixNSMLeafFrameFactory(BTreeTypeAwareTupleWriterFactory tupleWriterFactory) { this.tupleWriterFactory = tupleWriterFactory; } @@ -39,8 +38,7 @@ public class BTreeFieldPrefixNSMLeafFrameFactory implements ITreeIndexFrameFacto } @Override - public ITreeIndexTupleWriterFactory getTupleWriterFactory() { + public BTreeTypeAwareTupleWriterFactory getTupleWriterFactory() { return tupleWriterFactory; } - } http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMInteriorFrameFactory.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMInteriorFrameFactory.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMInteriorFrameFactory.java index 4ecb83d..4594f70 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMInteriorFrameFactory.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMInteriorFrameFactory.java @@ -21,15 +21,15 @@ package org.apache.hyracks.storage.am.btree.frames; import org.apache.hyracks.storage.am.btree.api.IBTreeInteriorFrame; import org.apache.hyracks.storage.am.common.api.ITreeIndexFrameFactory; -import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleWriterFactory; +import org.apache.hyracks.storage.am.common.tuples.TypeAwareTupleWriterFactory; public class BTreeNSMInteriorFrameFactory implements ITreeIndexFrameFactory { private static final long serialVersionUID = 1L; - private final ITreeIndexTupleWriterFactory tupleWriterFactory; + private final TypeAwareTupleWriterFactory tupleWriterFactory; - public BTreeNSMInteriorFrameFactory(ITreeIndexTupleWriterFactory tupleWriterFactory) { + public BTreeNSMInteriorFrameFactory(TypeAwareTupleWriterFactory tupleWriterFactory) { this.tupleWriterFactory = tupleWriterFactory; } @@ -39,7 +39,7 @@ public class BTreeNSMInteriorFrameFactory implements ITreeIndexFrameFactory { } @Override - public ITreeIndexTupleWriterFactory getTupleWriterFactory() { + public TypeAwareTupleWriterFactory getTupleWriterFactory() { return tupleWriterFactory; } http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMLeafFrame.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMLeafFrame.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMLeafFrame.java index 15251fa..09a4db4 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMLeafFrame.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMLeafFrame.java @@ -307,11 +307,6 @@ public class BTreeNSMLeafFrame extends TreeIndexNSMFrame implements IBTreeLeafFr } @Override - public ITreeIndexTupleReference createTupleReference() { - return tupleWriter.createTupleReference(); - } - - @Override public int findTupleIndex(ITupleReference searchKey, ITreeIndexTupleReference pageTuple, MultiComparator cmp, FindTupleMode ftm, FindTupleNoExactMatchPolicy ftp) throws HyracksDataException { return slotManager.findTupleIndex(searchKey, pageTuple, cmp, ftm, ftp); http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMLeafFrameFactory.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMLeafFrameFactory.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMLeafFrameFactory.java index 3cf0ba5..1ba9526 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMLeafFrameFactory.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMLeafFrameFactory.java @@ -19,28 +19,26 @@ package org.apache.hyracks.storage.am.btree.frames; -import org.apache.hyracks.storage.am.btree.api.IBTreeLeafFrame; import org.apache.hyracks.storage.am.common.api.ITreeIndexFrameFactory; -import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleWriterFactory; +import org.apache.hyracks.storage.am.common.tuples.TypeAwareTupleWriterFactory; public class BTreeNSMLeafFrameFactory implements ITreeIndexFrameFactory { private static final long serialVersionUID = 1L; - private final ITreeIndexTupleWriterFactory tupleWriterFactory; + private final TypeAwareTupleWriterFactory tupleWriterFactory; - public BTreeNSMLeafFrameFactory(ITreeIndexTupleWriterFactory tupleWriterFactory) { + public BTreeNSMLeafFrameFactory(TypeAwareTupleWriterFactory tupleWriterFactory) { this.tupleWriterFactory = tupleWriterFactory; } @Override - public IBTreeLeafFrame createFrame() { + public BTreeNSMLeafFrame createFrame() { return new BTreeNSMLeafFrame(tupleWriterFactory.createTupleWriter()); } @Override - public ITreeIndexTupleWriterFactory getTupleWriterFactory() { + public TypeAwareTupleWriterFactory getTupleWriterFactory() { return tupleWriterFactory; } - } http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTree.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTree.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTree.java index f781a88..7206fee 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTree.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTree.java @@ -39,6 +39,7 @@ import org.apache.hyracks.storage.am.btree.api.IBTreeLeafFrame; import org.apache.hyracks.storage.am.btree.api.ITupleAcceptor; import org.apache.hyracks.storage.am.btree.frames.BTreeNSMInteriorFrame; import org.apache.hyracks.storage.am.btree.impls.BTreeOpContext.PageValidationInfo; +import org.apache.hyracks.storage.am.common.api.IBTreeIndexTupleReference; import org.apache.hyracks.storage.am.common.api.IPageManager; import org.apache.hyracks.storage.am.common.api.ISplitKey; import org.apache.hyracks.storage.am.common.api.ITreeIndexAccessor; @@ -392,6 +393,8 @@ public class BTree extends AbstractTreeIndex { throws Exception { FrameOpSpaceStatus spaceStatus = ctx.getLeafFrame().hasSpaceUpdate(tuple, oldTupleIndex); ITupleReference beforeTuple = ctx.getLeafFrame().getMatchingKeyTuple(tuple, oldTupleIndex); + IBTreeIndexTupleReference beforeBTreeTuple = (IBTreeIndexTupleReference) beforeTuple; + ctx.getLeafFrame().getTupleWriter().setUpdated(beforeBTreeTuple.flipUpdated()); boolean restartOp = false; switch (spaceStatus) { case SUFFICIENT_INPLACE_SPACE: { @@ -437,6 +440,7 @@ public class BTree extends AbstractTreeIndex { throw new IllegalStateException("NYI: " + spaceStatus); } } + ctx.getLeafFrame().getTupleWriter().setUpdated(false); return restartOp; } @@ -755,7 +759,7 @@ public class BTree extends AbstractTreeIndex { } private BTreeOpContext createOpContext(IIndexAccessor accessor, IModificationOperationCallback modificationCallback, - ISearchOperationCallback searchCallback, int[] logTupleFields) { + ISearchOperationCallback searchCallback, int[] logTupleFields) { return new BTreeOpContext(accessor, leafFrameFactory, interiorFrameFactory, freePageManager, cmpFactories, modificationCallback, searchCallback, logTupleFields); } @@ -821,7 +825,7 @@ public class BTree extends AbstractTreeIndex { } public ITreeIndexAccessor createAccessor(IModificationOperationCallback modificationCallback, - ISearchOperationCallback searchCallback, int[] logTupleFields) { + ISearchOperationCallback searchCallback, int[] logTupleFields) { return new BTreeAccessor(this, modificationCallback, searchCallback, logTupleFields); } @@ -845,7 +849,7 @@ public class BTree extends AbstractTreeIndex { } public BTreeAccessor(BTree btree, IModificationOperationCallback modificationCalback, - ISearchOperationCallback searchCallback, int[] logTupleFields) { + ISearchOperationCallback searchCallback, int[] logTupleFields) { this.btree = btree; this.ctx = btree.createOpContext(this, modificationCalback, searchCallback, logTupleFields); } http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTreeFieldPrefixTupleReference.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTreeFieldPrefixTupleReference.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTreeFieldPrefixTupleReference.java new file mode 100644 index 0000000..54fec3d --- /dev/null +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTreeFieldPrefixTupleReference.java @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.hyracks.storage.am.btree.impls; + +import org.apache.hyracks.storage.am.btree.api.IPrefixSlotManager; +import org.apache.hyracks.storage.am.btree.frames.BTreeFieldPrefixNSMLeafFrame; +import org.apache.hyracks.storage.am.common.api.IBTreeIndexTupleReference; +import org.apache.hyracks.storage.am.common.api.ITreeIndexFrame; + +public class BTreeFieldPrefixTupleReference implements IBTreeIndexTupleReference { + + private final IBTreeIndexTupleReference helperTuple; + + private BTreeFieldPrefixNSMLeafFrame frame; + private int prefixTupleStartOff; + private int suffixTupleStartOff; + private int numPrefixFields; + private int fieldCount; + + public BTreeFieldPrefixTupleReference(IBTreeIndexTupleReference helperTuple) { + this.helperTuple = helperTuple; + this.fieldCount = helperTuple.getFieldCount(); + } + + @Override + public void resetByTupleIndex(ITreeIndexFrame frame, int tupleIndex) { + this.frame = (BTreeFieldPrefixNSMLeafFrame) frame; + IPrefixSlotManager slotManager = this.frame.getSlotManager(); + int tupleSlotOff = slotManager.getTupleSlotOff(tupleIndex); + int tupleSlot = this.frame.getBuffer().getInt(tupleSlotOff); + int prefixSlotNum = slotManager.decodeFirstSlotField(tupleSlot); + suffixTupleStartOff = slotManager.decodeSecondSlotField(tupleSlot); + + if (prefixSlotNum != FieldPrefixSlotManager.TUPLE_UNCOMPRESSED) { + int prefixSlotOff = slotManager.getPrefixSlotOff(prefixSlotNum); + int prefixSlot = this.frame.getBuffer().getInt(prefixSlotOff); + numPrefixFields = slotManager.decodeFirstSlotField(prefixSlot); + prefixTupleStartOff = slotManager.decodeSecondSlotField(prefixSlot); + } else { + numPrefixFields = 0; + prefixTupleStartOff = -1; + } + } + + @Override + public void setFieldCount(int fieldCount) { + this.fieldCount = fieldCount; + } + + @Override + public void setFieldCount(int fieldStartIndex, int fieldCount) { + throw new UnsupportedOperationException("Not supported."); + } + + @Override + public int getFieldCount() { + return fieldCount; + } + + @Override + public byte[] getFieldData(int fIdx) { + return frame.getBuffer().array(); + } + + @Override + public int getFieldLength(int fIdx) { + if (fIdx < numPrefixFields) { + helperTuple.setFieldCount(numPrefixFields); + helperTuple.resetByTupleOffset(frame.getBuffer().array(), prefixTupleStartOff); + return helperTuple.getFieldLength(fIdx); + } else { + helperTuple.setFieldCount(numPrefixFields, fieldCount - numPrefixFields); + helperTuple.resetByTupleOffset(frame.getBuffer().array(), suffixTupleStartOff); + return helperTuple.getFieldLength(fIdx - numPrefixFields); + } + } + + @Override + public int getFieldStart(int fIdx) { + if (fIdx < numPrefixFields) { + helperTuple.setFieldCount(numPrefixFields); + helperTuple.resetByTupleOffset(frame.getBuffer().array(), prefixTupleStartOff); + return helperTuple.getFieldStart(fIdx); + } else { + helperTuple.setFieldCount(numPrefixFields, fieldCount - numPrefixFields); + helperTuple.resetByTupleOffset(frame.getBuffer().array(), suffixTupleStartOff); + return helperTuple.getFieldStart(fIdx - numPrefixFields); + } + } + + // unsupported operation + @Override + public void resetByTupleOffset(byte[] buf, int tupleStartOffset) { + throw new UnsupportedOperationException("Resetting this type of frame by offset is not supported."); + } + + @Override + public int getTupleSize() { + return getSuffixTupleSize() + getPrefixTupleSize(); + } + + public int getSuffixTupleSize() { + helperTuple.setFieldCount(numPrefixFields, fieldCount - numPrefixFields); + helperTuple.resetByTupleOffset(frame.getBuffer().array(), suffixTupleStartOff); + return helperTuple.getTupleSize(); + } + + public int getPrefixTupleSize() { + if (numPrefixFields == 0) { + return 0; + } + helperTuple.setFieldCount(numPrefixFields); + helperTuple.resetByTupleOffset(frame.getBuffer().array(), prefixTupleStartOff); + return helperTuple.getTupleSize(); + } + + public int getNumPrefixFields() { + return numPrefixFields; + } + + @Override + public boolean flipUpdated() { + return helperTuple.flipUpdated(); + } + + @Override + public boolean isUpdated() { + return helperTuple.isUpdated(); + } +} http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/FieldPrefixTupleReference.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/FieldPrefixTupleReference.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/FieldPrefixTupleReference.java deleted file mode 100644 index 566947d..0000000 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/FieldPrefixTupleReference.java +++ /dev/null @@ -1,136 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.hyracks.storage.am.btree.impls; - -import org.apache.hyracks.storage.am.btree.api.IPrefixSlotManager; -import org.apache.hyracks.storage.am.btree.frames.BTreeFieldPrefixNSMLeafFrame; -import org.apache.hyracks.storage.am.common.api.ITreeIndexFrame; -import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleReference; - -public class FieldPrefixTupleReference implements ITreeIndexTupleReference { - - private final ITreeIndexTupleReference helperTuple; - - private BTreeFieldPrefixNSMLeafFrame frame; - private int prefixTupleStartOff; - private int suffixTupleStartOff; - private int numPrefixFields; - private int fieldCount; - - public FieldPrefixTupleReference(ITreeIndexTupleReference helperTuple) { - this.helperTuple = helperTuple; - this.fieldCount = helperTuple.getFieldCount(); - } - - @Override - public void resetByTupleIndex(ITreeIndexFrame frame, int tupleIndex) { - this.frame = (BTreeFieldPrefixNSMLeafFrame) frame; - IPrefixSlotManager slotManager = this.frame.getSlotManager(); - int tupleSlotOff = slotManager.getTupleSlotOff(tupleIndex); - int tupleSlot = this.frame.getBuffer().getInt(tupleSlotOff); - int prefixSlotNum = slotManager.decodeFirstSlotField(tupleSlot); - suffixTupleStartOff = slotManager.decodeSecondSlotField(tupleSlot); - - if (prefixSlotNum != FieldPrefixSlotManager.TUPLE_UNCOMPRESSED) { - int prefixSlotOff = slotManager.getPrefixSlotOff(prefixSlotNum); - int prefixSlot = this.frame.getBuffer().getInt(prefixSlotOff); - numPrefixFields = slotManager.decodeFirstSlotField(prefixSlot); - prefixTupleStartOff = slotManager.decodeSecondSlotField(prefixSlot); - } else { - numPrefixFields = 0; - prefixTupleStartOff = -1; - } - } - - @Override - public void setFieldCount(int fieldCount) { - this.fieldCount = fieldCount; - } - - @Override - public void setFieldCount(int fieldStartIndex, int fieldCount) { - throw new UnsupportedOperationException("Not supported."); - } - - @Override - public int getFieldCount() { - return fieldCount; - } - - @Override - public byte[] getFieldData(int fIdx) { - return frame.getBuffer().array(); - } - - @Override - public int getFieldLength(int fIdx) { - if (fIdx < numPrefixFields) { - helperTuple.setFieldCount(numPrefixFields); - helperTuple.resetByTupleOffset(frame.getBuffer().array(), prefixTupleStartOff); - return helperTuple.getFieldLength(fIdx); - } else { - helperTuple.setFieldCount(numPrefixFields, fieldCount - numPrefixFields); - helperTuple.resetByTupleOffset(frame.getBuffer().array(), suffixTupleStartOff); - return helperTuple.getFieldLength(fIdx - numPrefixFields); - } - } - - @Override - public int getFieldStart(int fIdx) { - if (fIdx < numPrefixFields) { - helperTuple.setFieldCount(numPrefixFields); - helperTuple.resetByTupleOffset(frame.getBuffer().array(), prefixTupleStartOff); - return helperTuple.getFieldStart(fIdx); - } else { - helperTuple.setFieldCount(numPrefixFields, fieldCount - numPrefixFields); - helperTuple.resetByTupleOffset(frame.getBuffer().array(), suffixTupleStartOff); - return helperTuple.getFieldStart(fIdx - numPrefixFields); - } - } - - // unsupported operation - @Override - public void resetByTupleOffset(byte[] buf, int tupleStartOffset) { - throw new UnsupportedOperationException("Resetting this type of frame by offset is not supported."); - } - - @Override - public int getTupleSize() { - return getSuffixTupleSize() + getPrefixTupleSize(); - } - - public int getSuffixTupleSize() { - helperTuple.setFieldCount(numPrefixFields, fieldCount - numPrefixFields); - helperTuple.resetByTupleOffset(frame.getBuffer().array(), suffixTupleStartOff); - return helperTuple.getTupleSize(); - } - - public int getPrefixTupleSize() { - if (numPrefixFields == 0) { - return 0; - } - helperTuple.setFieldCount(numPrefixFields); - helperTuple.resetByTupleOffset(frame.getBuffer().array(), prefixTupleStartOff); - return helperTuple.getTupleSize(); - } - - public int getNumPrefixFields() { - return numPrefixFields; - } -} http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/tuples/BTreeTypeAwareTupleReference.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/tuples/BTreeTypeAwareTupleReference.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/tuples/BTreeTypeAwareTupleReference.java new file mode 100644 index 0000000..5b89a19 --- /dev/null +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/tuples/BTreeTypeAwareTupleReference.java @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.hyracks.storage.am.btree.tuples; + +import org.apache.hyracks.api.dataflow.value.ITypeTraits; +import org.apache.hyracks.storage.am.common.api.IBTreeIndexTupleReference; +import org.apache.hyracks.storage.am.common.tuples.TypeAwareTupleReference; +import org.apache.hyracks.storage.am.common.util.BitOperationUtils; + +public class BTreeTypeAwareTupleReference extends TypeAwareTupleReference implements IBTreeIndexTupleReference { + + public static final byte UPDATE_BIT_OFFSET = 6; + + protected final boolean updateAware; + + public BTreeTypeAwareTupleReference(ITypeTraits[] typeTraits, boolean updateAware) { + super(typeTraits); + this.updateAware = updateAware; + } + + @Override + public boolean flipUpdated() { + if (updateAware) { + final byte mask = (byte) (1 << UPDATE_BIT_OFFSET); + if ((buf[tupleStartOff] & mask) == 0) { + buf[tupleStartOff] |= mask; + return true; + } else { + buf[tupleStartOff] &= ~mask; + return false; + } + } else { + return false; + } + } + + @Override + public boolean isUpdated() { + if (updateAware) { + return BitOperationUtils.getBit(buf, tupleStartOff, UPDATE_BIT_OFFSET); + } else { + return false; + } + } + +} http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/tuples/BTreeTypeAwareTupleWriter.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/tuples/BTreeTypeAwareTupleWriter.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/tuples/BTreeTypeAwareTupleWriter.java new file mode 100644 index 0000000..083462b --- /dev/null +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/tuples/BTreeTypeAwareTupleWriter.java @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.hyracks.storage.am.btree.tuples; + +import org.apache.hyracks.api.dataflow.value.ITypeTraits; +import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleWriter; +import org.apache.hyracks.storage.am.common.tuples.TypeAwareTupleWriter; + +public class BTreeTypeAwareTupleWriter extends TypeAwareTupleWriter implements ITreeIndexTupleWriter { + protected final boolean updateAware; + protected boolean isUpdated; + + public BTreeTypeAwareTupleWriter(ITypeTraits[] typeTraits, boolean updateAware) { + super(typeTraits); + this.updateAware = updateAware; + } + + @Override + public BTreeTypeAwareTupleReference createTupleReference() { + return new BTreeTypeAwareTupleReference(typeTraits, updateAware); + } + + @Override + public void setUpdated(boolean isUpdated) { + this.isUpdated = isUpdated; + } +} http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/tuples/BTreeTypeAwareTupleWriterFactory.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/tuples/BTreeTypeAwareTupleWriterFactory.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/tuples/BTreeTypeAwareTupleWriterFactory.java new file mode 100644 index 0000000..7089e79 --- /dev/null +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/tuples/BTreeTypeAwareTupleWriterFactory.java @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.hyracks.storage.am.btree.tuples; + +import org.apache.hyracks.api.dataflow.value.ITypeTraits; +import org.apache.hyracks.storage.am.common.tuples.TypeAwareTupleWriterFactory; + +public class BTreeTypeAwareTupleWriterFactory extends TypeAwareTupleWriterFactory { + + private static final long serialVersionUID = 1L; + protected final boolean updateAware; + + public BTreeTypeAwareTupleWriterFactory(ITypeTraits[] typeTraits, boolean updateAware) { + super(typeTraits); + this.updateAware = updateAware; + } + + @Override + public BTreeTypeAwareTupleWriter createTupleWriter() { + return new BTreeTypeAwareTupleWriter(typeTraits, updateAware); + } +} http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/util/BTreeUtils.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/util/BTreeUtils.java b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/util/BTreeUtils.java index 303777e..7f992f9 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/util/BTreeUtils.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/util/BTreeUtils.java @@ -29,18 +29,22 @@ import org.apache.hyracks.storage.am.btree.frames.BTreeLeafFrameType; import org.apache.hyracks.storage.am.btree.frames.BTreeNSMInteriorFrameFactory; import org.apache.hyracks.storage.am.btree.frames.BTreeNSMLeafFrameFactory; import org.apache.hyracks.storage.am.btree.impls.BTree; +import org.apache.hyracks.storage.am.btree.tuples.BTreeTypeAwareTupleWriterFactory; import org.apache.hyracks.storage.am.common.api.IPageManager; import org.apache.hyracks.storage.am.common.api.ITreeIndexFrameFactory; -import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleWriterFactory; -import org.apache.hyracks.storage.am.common.tuples.TypeAwareTupleWriterFactory; import org.apache.hyracks.storage.common.MultiComparator; import org.apache.hyracks.storage.common.buffercache.IBufferCache; public class BTreeUtils { + + private BTreeUtils() { + } + public static BTree createBTree(IBufferCache bufferCache, ITypeTraits[] typeTraits, IBinaryComparatorFactory[] cmpFactories, BTreeLeafFrameType leafType, FileReference file, - IPageManager freePageManager) throws HyracksDataException { - TypeAwareTupleWriterFactory tupleWriterFactory = new TypeAwareTupleWriterFactory(typeTraits); + IPageManager freePageManager, boolean updateAware) throws HyracksDataException { + BTreeTypeAwareTupleWriterFactory tupleWriterFactory = + new BTreeTypeAwareTupleWriterFactory(typeTraits, updateAware); ITreeIndexFrameFactory leafFrameFactory = getLeafFrameFactory(tupleWriterFactory, leafType); ITreeIndexFrameFactory interiorFrameFactory = new BTreeNSMInteriorFrameFactory(tupleWriterFactory); return new BTree(bufferCache, freePageManager, interiorFrameFactory, leafFrameFactory, cmpFactories, @@ -48,9 +52,10 @@ public class BTreeUtils { } public static BTree createBTree(IBufferCache bufferCache, IPageManager freePageManager, ITypeTraits[] typeTraits, - IBinaryComparatorFactory[] cmpFactories, BTreeLeafFrameType leafType, FileReference file) - throws HyracksDataException { - TypeAwareTupleWriterFactory tupleWriterFactory = new TypeAwareTupleWriterFactory(typeTraits); + IBinaryComparatorFactory[] cmpFactories, BTreeLeafFrameType leafType, FileReference file, + boolean updateAware) throws HyracksDataException { + BTreeTypeAwareTupleWriterFactory tupleWriterFactory = + new BTreeTypeAwareTupleWriterFactory(typeTraits, updateAware); ITreeIndexFrameFactory leafFrameFactory = getLeafFrameFactory(tupleWriterFactory, leafType); ITreeIndexFrameFactory interiorFrameFactory = new BTreeNSMInteriorFrameFactory(tupleWriterFactory); return new BTree(bufferCache, freePageManager, interiorFrameFactory, leafFrameFactory, cmpFactories, @@ -70,7 +75,7 @@ public class BTreeUtils { return new MultiComparator(newCmps); } - public static ITreeIndexFrameFactory getLeafFrameFactory(ITreeIndexTupleWriterFactory tupleWriterFactory, + public static ITreeIndexFrameFactory getLeafFrameFactory(BTreeTypeAwareTupleWriterFactory tupleWriterFactory, BTreeLeafFrameType leafType) throws HyracksDataException { switch (leafType) { case REGULAR_NSM: { http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/IBTreeIndexTupleReference.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/IBTreeIndexTupleReference.java b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/IBTreeIndexTupleReference.java new file mode 100644 index 0000000..7346695 --- /dev/null +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/IBTreeIndexTupleReference.java @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.hyracks.storage.am.common.api; + +public interface IBTreeIndexTupleReference extends ITreeIndexTupleReference { + + /** + * @return Method returns if update-in-place bit is set for a BTree tuple + */ + boolean isUpdated(); + + /** + * Method changes the value of update-in-place bit from 0 to 1, or from 1 to 0 + * + * @return New value of update-in-place bit + */ + boolean flipUpdated(); +} http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexFrameFactory.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexFrameFactory.java b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexFrameFactory.java index 8a705dc..007c84f 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexFrameFactory.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexFrameFactory.java @@ -22,5 +22,6 @@ import java.io.Serializable; public interface ITreeIndexFrameFactory extends Serializable { ITreeIndexFrame createFrame(); + ITreeIndexTupleWriterFactory getTupleWriterFactory(); } http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexTupleWriter.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexTupleWriter.java b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexTupleWriter.java index b439a97..dfebe52 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexTupleWriter.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexTupleWriter.java @@ -43,4 +43,14 @@ public interface ITreeIndexTupleWriter { // in the LSM-BTree can be either matter or anti-matter tuples and we want // to calculate the size of all tuples in the frame. public int getCopySpaceRequired(ITupleReference tuple); + + /** + * Method sets tuple writer to raise update-in-place bit + * + * @param isUpdated + * Value of update-in-place bit + */ + default void setUpdated(boolean isUpdated) { + // Dummy default: most tuple writers don't support update bit + } } http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexTupleWriterFactory.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexTupleWriterFactory.java b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexTupleWriterFactory.java index 22bff98..1bc807c 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexTupleWriterFactory.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexTupleWriterFactory.java @@ -21,6 +21,7 @@ package org.apache.hyracks.storage.am.common.api; import java.io.Serializable; +@FunctionalInterface public interface ITreeIndexTupleWriterFactory extends Serializable { - public ITreeIndexTupleWriter createTupleWriter(); + ITreeIndexTupleWriter createTupleWriter(); } http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/frames/TreeIndexNSMFrame.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/frames/TreeIndexNSMFrame.java b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/frames/TreeIndexNSMFrame.java index 0948704..5f001e0 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/frames/TreeIndexNSMFrame.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/frames/TreeIndexNSMFrame.java @@ -186,8 +186,8 @@ public abstract class TreeIndexNSMFrame implements ITreeIndexFrame { // maintain space information buf.putInt(Constants.TUPLE_COUNT_OFFSET, buf.getInt(Constants.TUPLE_COUNT_OFFSET) - 1); - buf.putInt(TOTAL_FREE_SPACE_OFFSET, buf.getInt(TOTAL_FREE_SPACE_OFFSET) + tupleSize + slotManager - .getSlotSize()); + buf.putInt(TOTAL_FREE_SPACE_OFFSET, + buf.getInt(TOTAL_FREE_SPACE_OFFSET) + tupleSize + slotManager.getSlotSize()); } @Override @@ -243,8 +243,8 @@ public abstract class TreeIndexNSMFrame implements ITreeIndexFrame { int bytesWritten = tupleWriter.writeTuple(tuple, buf.array(), buf.getInt(Constants.FREE_SPACE_OFFSET)); buf.putInt(Constants.TUPLE_COUNT_OFFSET, buf.getInt(Constants.TUPLE_COUNT_OFFSET) + 1); buf.putInt(Constants.FREE_SPACE_OFFSET, buf.getInt(Constants.FREE_SPACE_OFFSET) + bytesWritten); - buf.putInt(TOTAL_FREE_SPACE_OFFSET, buf.getInt(TOTAL_FREE_SPACE_OFFSET) - bytesWritten - slotManager - .getSlotSize()); + buf.putInt(TOTAL_FREE_SPACE_OFFSET, + buf.getInt(TOTAL_FREE_SPACE_OFFSET) - bytesWritten - slotManager.getSlotSize()); } @Override http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java index 0c2081a..609c51a 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java @@ -22,6 +22,7 @@ package org.apache.hyracks.storage.am.common.tuples; import org.apache.hyracks.data.std.primitive.ShortPointable; import org.apache.hyracks.storage.am.common.api.ITreeIndexFrame; import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleReference; +import org.apache.hyracks.storage.am.common.util.BitOperationUtils; public class SimpleTupleReference implements ITreeIndexTupleReference { @@ -88,7 +89,7 @@ public class SimpleTupleReference implements ITreeIndexTupleReference { } protected int getNullFlagsBytes() { - return (int) Math.ceil(fieldCount / 8.0); + return BitOperationUtils.getFlagBytes(fieldCount); } protected int getFieldSlotsBytes() { http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java index 7aaa983..410a0e3 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java @@ -23,6 +23,7 @@ import java.nio.ByteBuffer; import org.apache.hyracks.dataflow.common.data.accessors.ITupleReference; import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleWriter; +import org.apache.hyracks.storage.am.common.util.BitOperationUtils; /* * This class should be replaced by a Util class @@ -110,7 +111,7 @@ public class SimpleTupleWriter implements ITreeIndexTupleWriter { } protected int getNullFlagsBytes(ITupleReference tuple) { - return (int) Math.ceil(tuple.getFieldCount() / 8.0); + return BitOperationUtils.getFlagBytes(tuple.getFieldCount()); } protected int getFieldSlotsBytes(ITupleReference tuple) { @@ -118,7 +119,7 @@ public class SimpleTupleWriter implements ITreeIndexTupleWriter { } protected int getNullFlagsBytes(ITupleReference tuple, int startField, int numFields) { - return (int) Math.ceil(numFields / 8.0); + return BitOperationUtils.getFlagBytes(numFields); } protected int getFieldSlotsBytes(ITupleReference tuple, int startField, int numFields) { http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleReference.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleReference.java b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleReference.java index e278f6c..11f86c9 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleReference.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleReference.java @@ -22,6 +22,7 @@ package org.apache.hyracks.storage.am.common.tuples; import org.apache.hyracks.api.dataflow.value.ITypeTraits; import org.apache.hyracks.storage.am.common.api.ITreeIndexFrame; import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleReference; +import org.apache.hyracks.storage.am.common.util.BitOperationUtils; import org.apache.hyracks.util.encoding.VarLenIntEncoderDecoder; import org.apache.hyracks.util.encoding.VarLenIntEncoderDecoder.VarLenIntDecoder; @@ -119,7 +120,7 @@ public class TypeAwareTupleReference implements ITreeIndexTupleReference { } protected int getNullFlagsBytes() { - return (int) Math.ceil(fieldCount / 8.0); + return BitOperationUtils.getFlagBytes(fieldCount); } @Override http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleWriter.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleWriter.java b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleWriter.java index 1d55d4d..eb6fe2a 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleWriter.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleWriter.java @@ -23,8 +23,8 @@ import java.nio.ByteBuffer; import org.apache.hyracks.api.dataflow.value.ITypeTraits; import org.apache.hyracks.dataflow.common.data.accessors.ITupleReference; -import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleReference; import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleWriter; +import org.apache.hyracks.storage.am.common.util.BitOperationUtils; import org.apache.hyracks.util.encoding.VarLenIntEncoderDecoder; public class TypeAwareTupleWriter implements ITreeIndexTupleWriter { @@ -55,7 +55,7 @@ public class TypeAwareTupleWriter implements ITreeIndexTupleWriter { } @Override - public ITreeIndexTupleReference createTupleReference() { + public TypeAwareTupleReference createTupleReference() { return new TypeAwareTupleReference(typeTraits); } @@ -114,7 +114,7 @@ public class TypeAwareTupleWriter implements ITreeIndexTupleWriter { } protected int getNullFlagsBytes(ITupleReference tuple) { - return (int) Math.ceil(tuple.getFieldCount() / 8.0); + return BitOperationUtils.getFlagBytes(tuple.getFieldCount()); } protected int getFieldSlotsBytes(ITupleReference tuple) { @@ -128,7 +128,7 @@ public class TypeAwareTupleWriter implements ITreeIndexTupleWriter { } protected int getNullFlagsBytes(int numFields) { - return (int) Math.ceil(numFields / 8.0); + return BitOperationUtils.getFlagBytes(numFields); } protected int getFieldSlotsBytes(ITupleReference tuple, int startField, int numFields) { http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleWriterFactory.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleWriterFactory.java b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleWriterFactory.java index 741a53a..a0fa505 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleWriterFactory.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/TypeAwareTupleWriterFactory.java @@ -20,11 +20,9 @@ package org.apache.hyracks.storage.am.common.tuples; import org.apache.hyracks.api.dataflow.value.ITypeTraits; -import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleWriter; import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleWriterFactory; -public class TypeAwareTupleWriterFactory implements - ITreeIndexTupleWriterFactory { +public class TypeAwareTupleWriterFactory implements ITreeIndexTupleWriterFactory { private static final long serialVersionUID = 1L; protected ITypeTraits[] typeTraits; @@ -34,7 +32,7 @@ public class TypeAwareTupleWriterFactory implements } @Override - public ITreeIndexTupleWriter createTupleWriter() { + public TypeAwareTupleWriter createTupleWriter() { return new TypeAwareTupleWriter(typeTraits); } http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/util/BitOperationUtils.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/util/BitOperationUtils.java b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/util/BitOperationUtils.java new file mode 100644 index 0000000..9d0d229 --- /dev/null +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/util/BitOperationUtils.java @@ -0,0 +1,38 @@ +/* + + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + + */ +package org.apache.hyracks.storage.am.common.util; + +public class BitOperationUtils { + private BitOperationUtils() { + } + + public static void setBit(byte[] targetBuf, int targetOff, byte bitPosition) { + targetBuf[targetOff] = (byte) (targetBuf[targetOff] | (1 << bitPosition)); + } + + public static boolean getBit(byte[] targetBuf, int targetOff, byte bitPosition) { + return (byte) (targetBuf[targetOff] & (1 << bitPosition)) != 0; + } + + public static int getFlagBytes(int numFlags) { + return (int) Math.ceil((double) numFlags / 8.0); + } +} http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/dataflow/LSMBTreeLocalResource.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/dataflow/LSMBTreeLocalResource.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/dataflow/LSMBTreeLocalResource.java index f6e311c..c283502 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/dataflow/LSMBTreeLocalResource.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/dataflow/LSMBTreeLocalResource.java @@ -70,11 +70,13 @@ public class LSMBTreeLocalResource extends LsmResource { IIOManager ioManager = serviceCtx.getIoManager(); FileReference file = ioManager.resolve(path); List vbcs = vbcProvider.getVirtualBufferCaches(serviceCtx, file); + //TODO: enable updateAwareness for secondary LSMBTree indexes + boolean updateAware = false; return LSMBTreeUtil.createLSMTree(ioManager, vbcs, file, storageManager.getBufferCache(serviceCtx), typeTraits, cmpFactories, bloomFilterKeyFields, bloomFilterFalsePositiveRate, mergePolicyFactory.createMergePolicy(mergePolicyProperties, serviceCtx), opTrackerProvider.getOperationTracker(serviceCtx), ioSchedulerProvider.getIoScheduler(serviceCtx), ioOpCallbackFactory.createIoOpCallback(), isPrimary, filterTypeTraits, filterCmpFactories, btreeFields, - filterFields, durable, metadataPageManagerFactory); + filterFields, durable, metadataPageManagerFactory, updateAware); } } http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMBTree.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMBTree.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMBTree.java index 88f58d1..24b408c 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMBTree.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMBTree.java @@ -90,6 +90,7 @@ public class LSMBTree extends AbstractLSMIndex implements ITreeIndex { protected final ITreeIndexFrameFactory insertLeafFrameFactory; protected final ITreeIndexFrameFactory deleteLeafFrameFactory; protected final IBinaryComparatorFactory[] cmpFactories; + private final boolean updateAware; private final boolean needKeyDupCheck; @@ -105,13 +106,14 @@ public class LSMBTree extends AbstractLSMIndex implements ITreeIndex { double bloomFilterFalsePositiveRate, int fieldCount, IBinaryComparatorFactory[] cmpFactories, ILSMMergePolicy mergePolicy, ILSMOperationTracker opTracker, ILSMIOOperationScheduler ioScheduler, ILSMIOOperationCallback ioOpCallback, boolean needKeyDupCheck, int[] btreeFields, int[] filterFields, - boolean durable) throws HyracksDataException { + boolean durable, boolean updateAware) throws HyracksDataException { super(ioManager, virtualBufferCaches, diskBTreeFactory.getBufferCache(), fileManager, bloomFilterFalsePositiveRate, mergePolicy, opTracker, ioScheduler, ioOpCallback, filterFrameFactory, filterManager, filterFields, durable, filterHelper, btreeFields); this.insertLeafFrameFactory = insertLeafFrameFactory; this.deleteLeafFrameFactory = deleteLeafFrameFactory; this.cmpFactories = cmpFactories; + this.updateAware = updateAware; int i = 0; for (IVirtualBufferCache virtualBufferCache : virtualBufferCaches) { LSMBTreeMemoryComponent mutableComponent = new LSMBTreeMemoryComponent( @@ -145,6 +147,7 @@ public class LSMBTree extends AbstractLSMIndex implements ITreeIndex { this.cmpFactories = cmpFactories; this.needKeyDupCheck = needKeyDupCheck; this.hasBloomFilter = true; + this.updateAware = false; componentFactory = new LSMBTreeDiskComponentFactory(diskBTreeFactory, bloomFilterFactory, null); bulkLoadComponentFactory = new LSMBTreeDiskComponentFactory(bulkLoadBTreeFactory, bloomFilterFactory, null); } @@ -325,6 +328,11 @@ public class LSMBTree extends AbstractLSMIndex implements ITreeIndex { try { while (scanCursor.hasNext()) { scanCursor.next(); + // we can safely throw away updated tuples in secondary BTree components, because they correspond to + // deleted tuples + if (updateAware && ((LSMBTreeTupleReference) scanCursor.getTuple()).isUpdated()) { + continue; + } componentBulkLoader.add(scanCursor.getTuple()); } } finally { http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeCopyTupleWriter.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeCopyTupleWriter.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeCopyTupleWriter.java index 4e814d0..44002ae 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeCopyTupleWriter.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeCopyTupleWriter.java @@ -23,9 +23,9 @@ import org.apache.hyracks.api.dataflow.value.ITypeTraits; import org.apache.hyracks.dataflow.common.data.accessors.ITupleReference; public class LSMBTreeCopyTupleWriter extends LSMBTreeTupleWriter { - public LSMBTreeCopyTupleWriter(ITypeTraits[] typeTraits, int numKeyFields){ + public LSMBTreeCopyTupleWriter(ITypeTraits[] typeTraits, int numKeyFields, boolean updateAware) { // Third parameter is never used locally, just give false. - super(typeTraits, numKeyFields, false); + super(typeTraits, numKeyFields, false, updateAware); } @Override http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeCopyTupleWriterFactory.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeCopyTupleWriterFactory.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeCopyTupleWriterFactory.java index 928b3fa..02cd900 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeCopyTupleWriterFactory.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeCopyTupleWriterFactory.java @@ -20,22 +20,20 @@ package org.apache.hyracks.storage.am.lsm.btree.tuples; import org.apache.hyracks.api.dataflow.value.ITypeTraits; -import org.apache.hyracks.storage.am.common.api.ITreeIndexTupleWriter; -import org.apache.hyracks.storage.am.common.tuples.TypeAwareTupleWriterFactory; +import org.apache.hyracks.storage.am.btree.tuples.BTreeTypeAwareTupleWriter; +import org.apache.hyracks.storage.am.btree.tuples.BTreeTypeAwareTupleWriterFactory; -public class LSMBTreeCopyTupleWriterFactory extends TypeAwareTupleWriterFactory { +public class LSMBTreeCopyTupleWriterFactory extends BTreeTypeAwareTupleWriterFactory { private static final long serialVersionUID = 1L; - private final ITypeTraits[] typeTraits; private final int numKeyFields; - public LSMBTreeCopyTupleWriterFactory(ITypeTraits[] typeTraits, int numKeyFields) { - super(typeTraits); - this.typeTraits = typeTraits; + public LSMBTreeCopyTupleWriterFactory(ITypeTraits[] typeTraits, int numKeyFields, boolean updateAware) { + super(typeTraits, updateAware); this.numKeyFields = numKeyFields; } @Override - public ITreeIndexTupleWriter createTupleWriter() { - return new LSMBTreeCopyTupleWriter(typeTraits, numKeyFields); + public BTreeTypeAwareTupleWriter createTupleWriter() { + return new LSMBTreeCopyTupleWriter(typeTraits, numKeyFields, updateAware); } } http://git-wip-us.apache.org/repos/asf/asterixdb/blob/e66346a3/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeTupleReference.java ---------------------------------------------------------------------- diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeTupleReference.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeTupleReference.java index 73aa321..2a5e232 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeTupleReference.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/tuples/LSMBTreeTupleReference.java @@ -20,11 +20,12 @@ package org.apache.hyracks.storage.am.lsm.btree.tuples; import org.apache.hyracks.api.dataflow.value.ITypeTraits; +import org.apache.hyracks.storage.am.btree.tuples.BTreeTypeAwareTupleReference; import org.apache.hyracks.storage.am.common.api.ITreeIndexFrame; -import org.apache.hyracks.storage.am.common.tuples.TypeAwareTupleReference; +import org.apache.hyracks.storage.am.common.util.BitOperationUtils; import org.apache.hyracks.storage.am.lsm.common.api.ILSMTreeTupleReference; -public class LSMBTreeTupleReference extends TypeAwareTupleReference implements ILSMTreeTupleReference { +public class LSMBTreeTupleReference extends BTreeTypeAwareTupleReference implements ILSMTreeTupleReference { // Indicates whether the last call to setFieldCount() was initiated by // by the outside or whether it was called internally to set up an @@ -32,8 +33,8 @@ public class LSMBTreeTupleReference extends TypeAwareTupleReference implements I private boolean resetFieldCount = false; private final int numKeyFields; - public LSMBTreeTupleReference(ITypeTraits[] typeTraits, int numKeyFields) { - super(typeTraits); + public LSMBTreeTupleReference(ITypeTraits[] typeTraits, int numKeyFields, boolean updateAware) { + super(typeTraits, updateAware); this.numKeyFields = numKeyFields; } @@ -76,18 +77,18 @@ public class LSMBTreeTupleReference extends TypeAwareTupleReference implements I @Override protected int getNullFlagsBytes() { - // +1.0 is for matter/antimatter bit. - return (int) Math.ceil((fieldCount + 1.0) / 8.0); + // number of fields + matter/antimatter bit. + int numBits = fieldCount + 1; + if (updateAware) { + numBits++; + } + return BitOperationUtils.getFlagBytes(numBits); } @Override public boolean isAntimatter() { - // Check if the leftmost bit is 0 or 1. - final byte mask = (byte) (1 << 7); - if ((buf[tupleStartOff] & mask) != 0) { - return true; - } - return false; + // Check antimatter bit. + return BitOperationUtils.getBit(buf, tupleStartOff, ANTIMATTER_BIT_OFFSET); } public int getTupleStart() {