asterixdb-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luo Chen (Code Review)" <do-not-re...@asterixdb.incubator.apache.org>
Subject Change in asterixdb[master]: [ASTERIXDB-2339] Add a new inverted index merge cursor
Date Fri, 23 Mar 2018 21:33:25 GMT
Luo Chen has posted comments on this change.

Change subject: [ASTERIXDB-2339] Add a new inverted index merge cursor
......................................................................


Patch Set 5:

(3 comments)

https://asterix-gerrit.ics.uci.edu/#/c/2519/3/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/impls/LSMInvertedIndexMergeCursor.java
File hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/impls/LSMInvertedIndexMergeCursor.java:

PS3, Line 242:    protected void checkKeyQueue() throws HyracksDataException {
             :         while (!keyQueue.isEmpty() || needPushElementIntoKeyQueue) {
             :             if (!keyQueue.isEmpty()) {
             :                 PriorityQueueElement checkElement = keyQueue.peek();
             :                 // If there is no previous tuple or the previous tuple can
be ignored
             :                 if (outputKeyElement == null) {
             :                     if (isDeleted(checkElement)) {
             :                         // If the key has been deleted then pop it and set
needPush to true.
             :                         // We cannot push immediately because the tuple may
be
             :                         // modified if hasNext() is called
             :                         outputKeyElement = checkElement;
             :                         needPushElementIntoKeyQueue = true;
             :                     } else {
             :                         // we have found the next record
             :                         return;
             :                     }
             :                 } else {
             :                     // Compare the previous tuple and the head tuple in the
PQ
             :                     if (keyCmp.compare(outputKeyElement.getTuple(), checkElement.getTuple())
== 0) {
             :                         // If the previous tuple and the head tuple are
             :                         // identical
             :                         // then pop the head tuple and push the next tuple
from
             :                         // the tree of head tuple
             : 
             :                         // the head element of PQ is useless now
             :                         PriorityQueueElement e = keyQueue.poll();
             :                         pushIntoKeyQueueAndReplace(e);
             :                     } else {
             :                         // If the previous tuple and the head tuple are different
             :                         // the info of previous tuple is useless
             :                         if (needPushElementIntoKeyQueue) {
             :                             pushIntoKeyQueueAndReplace(outputKeyElement);
             :                             needPushElementIntoKeyQueue = false;
             :                         }
             :                         outputKeyElement = null;
             :                     }
             :                 }
             :             } else {
             :                 // the priority queue is empty and needPush
             :                 // NOSONAR: outputKeyElement is not null when needPushElementIntoKeyQueue
= true
             :                 pushIntoKeyQueueAndReplace(outputKeyElement);
             :                 needPushElementIntoKeyQueue = false;
             :                 outputKeyElement = null;
             :             }
             :      
> no way to deduplicate this from checkPriorityQueue()?
You mean from LSMIndexSearchCursor? These two cursors are inherently different, and this cursor
uses two queues. Thus, I created a new cursor for this purpose.


https://asterix-gerrit.ics.uci.edu/#/c/2519/3/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/impls/LSMInvertedIndexRangeSearchCursor.java
File hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/impls/LSMInvertedIndexRangeSearchCursor.java:

PS3, Line 117: doClose(
> what's the rationale behind changing this method?
This was a mistake...addressed in the new patch


https://asterix-gerrit.ics.uci.edu/#/c/2519/3/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/ondisk/OnDiskInvertedIndex.java
File hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/ondisk/OnDiskInvertedIndex.java:

PS3, Line 319:            throw new IllegalStateException(
             :                             "Failed to append element to inverted list after
switching to a new page.");
             :                 }
             :             }
             :         }
> Is there some other way to do this besides instanceof?
Did some refactoring and created a specialized bulkloader for merge. This change could also
benefit future optimizations for merge/flush bulk loading...


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2519
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I57d039cd7e08033884529a204bff9acffd96d9bb
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen <cluo8@uci.edu>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Ian Maxon <imaxon@apache.org>
Gerrit-Reviewer: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Luo Chen <cluo8@uci.edu>
Gerrit-Reviewer: Taewoo Kim <taewok2@uci.edu>
Gerrit-Reviewer: abdullah alamoudi <bamousaa@gmail.com>
Gerrit-HasComments: Yes

Mime
View raw message