From notifications-return-158004-archive-asf-public=cust-asf.ponee.io@asterixdb.apache.org Wed Jun 2 20:18:09 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id A2A94180638 for ; Wed, 2 Jun 2021 22:18:09 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id E43F05FE79 for ; Wed, 2 Jun 2021 20:18:08 +0000 (UTC) Received: (qmail 60758 invoked by uid 500); 2 Jun 2021 20:18:08 -0000 Mailing-List: contact notifications-help@asterixdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.apache.org Delivered-To: mailing list notifications@asterixdb.apache.org Received: (qmail 60741 invoked by uid 99); 2 Jun 2021 20:18:07 -0000 Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2021 20:18:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id BBD4E1FF481 for ; Wed, 2 Jun 2021 20:18:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: 0.213 X-Spam-Level: X-Spam-Status: No, score=0.213 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=0.2, SPF_NONE=0.001, T_KAM_HTML_FONT_INVALID=0.01, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-ec2-va.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id On1Fdr3uCvW3 for ; Wed, 2 Jun 2021 20:18:05 +0000 (UTC) Received-SPF: None (mailfrom) identity=mailfrom; client-ip=128.195.1.133; helo=adrian-monk-v3.ics.uci.edu; envelope-from=do-not-reply@asterix-gerrit.ics.uci.edu; receiver= Received: from adrian-monk-v3.ics.uci.edu (adrian-monk-v3.ics.uci.edu [128.195.1.133]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 86314BD221 for ; Wed, 2 Jun 2021 20:18:03 +0000 (UTC) Received: from 8af6fa8c6e8b (vitalstatistix.ics.uci.edu [128.195.52.38]) by adrian-monk-v3.ics.uci.edu (Postfix) with ESMTP id 9D32CC02E302; Wed, 2 Jun 2021 13:17:58 -0700 (PDT) X-Gerrit-PatchSet: 1 Date: Wed, 2 Jun 2021 20:17:58 +0000 From: AsterixDB Code Review To: Till Westmann Message-ID: Auto-Submitted: auto-generated X-Gerrit-MessageType: newchange Subject: Change in asterixdb[master]: PLEASE EDIT to provide a meaningful commit message! X-Gerrit-Change-Id: I328bf4e80037a23b6d44139548436f6c4f232bba X-Gerrit-Change-Number: 11723 X-Gerrit-Project: asterixdb X-Gerrit-ChangeURL: X-Gerrit-Commit: af45f20bed9279b5c0f94fab3f243fde7e8e9e9e References: Reply-To: ggalvizo@uci.edu, dmitry.lychagin@couchbase.com, notifications@asterixdb.apache.org, lwhaywhu@gmail.com, tillw@apache.org MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/3.0.12 Content-Type: multipart/alternative; boundary="OsH6RUtJxCQ="; charset=UTF-8 X-ICS-MailScanner-Information: Please send mail to helpdesk@ics.uci.edu or more information X-ICS-MailScanner-ID: 9D32CC02E302.A3872 X-ICS-MailScanner: Found to be clean X-ICS-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-0.986, required 5, ALL_TRUSTED -1.00, FSL_HELO_NON_FQDN_1 0.00, HELO_NO_DOMAIN 0.00, HTML_MESSAGE 0.00, T_KAM_HTML_FONT_INVALID 0.01, URIBL_BLOCKED 0.00) X-ICS-MailScanner-From: do-not-reply@asterix-gerrit.ics.uci.edu --OsH6RUtJxCQ= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable From Glenn Galvizo : Glenn Galvizo has uploaded this c= hange for review=2E ( https://asterix-gerrit=2Eics=2Euci=2Eedu/c/asterixdb/= +/11723 ) Change subject: PLEASE EDIT to provide a meaningful commit mess= age! =2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E= =2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E= =2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E PLEASE = EDIT to provide a meaningful commit message! The following commits from yo= ur working branch will be included: commit 8cb9a782df3ec1c8e42a59341d7d7b3= 7d0eb5906 Author: ggalvizo Date: Sat May 29 19:43:34= 2021 -0700 Currently using the try-catch pattern to implement the imp= licit DISTINCT for array index maintenance=2E Change-Id: I328bf4e80037a23b= 6d44139548436f6c4f232bba --- M asterixdb/asterix-algebra/src/main/java/org/= apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule=2Eja= va M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/ope= rators/LSMSecondaryInsertDeleteWithNestedPlanOperatorNodePushable=2Ejava M = asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operator= s/LSMSecondaryUpsertWithNestedPlanOperatorNodePushable=2Ejava 3 files chang= ed, 66 insertions(+), 49 deletions(-) git pull ssh://asterix-gerrit=2E= ics=2Euci=2Eedu:29418/asterixdb refs/changes/23/11723/1 diff --git a/aster= ixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/Intro= duceSecondaryIndexInsertDeleteRule=2Ejava b/asterixdb/asterix-algebra/src/m= ain/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDe= leteRule=2Ejava index c3859be=2E=2E182687d 100644 --- a/asterixdb/asterix-a= lgebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryI= ndexInsertDeleteRule=2Ejava +++ b/asterixdb/asterix-algebra/src/main/java/o= rg/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule= =2Ejava @@ -449,7 +449,7 @@ context=2EcomputeAndSe= tTypeEnvironmentForOperator(unnestSourceOp); Unnes= tBranchCreator unnestSIDXBranch =3D buildUnnestBranch(unnestSourceOp, index= , newRecordVar, newMetaVar, recType, metaT= ype, dataset=2EhasMetaPart()); - unnestSIDXBranch=2E= applyProjectDistinct(); + unnestSIDXBranch=2EapplyPr= ojectOnly(); // If there exists a filter express= ion, add it to the top of our nested plan=2E filte= rExpression =3D (primaryIndexModificationOp=2EgetOperation() =3D=3D Kind=2E= UPSERT) ? null @@ -477,7 +477,7 @@ UnnestBranc= hCreator unnestBeforeSIDXBranch =3D buildUnnestBranch(unnestBeforeSourceOp,= index, primaryIndexModificationOp=2Eg= etBeforeOpRecordVar(), beforeOpMetaVar, recType, = metaType, dataset=2EhasMetaPart()); - = unnestBeforeSIDXBranch=2EapplyProjectDistinct(); + = unnestBeforeSIDXBranch=2EapplyProjectOnly(); = indexUpdate=2EgetNestedPlans()=2Eadd(unnestBeforeSIDXBranch=2EbuildBranc= h()); } } else if (index=2Ege= tIndexType() =3D=3D IndexType=2EARRAY && isBulkload) { @@ -1010,6 +1010,13 = @@ return varRef; } + public final void appl= yProjectOnly() throws AlgebricksException { + List projectVars =3D new ArrayList<>(this=2ElastFieldVars); + Pro= jectOperator projectOperator =3D new ProjectOperator(projectVars); + = projectOperator=2EsetSourceLocation(sourceLoc); + this=2Ecu= rrentTop =3D introduceNewOp(currentTop, projectOperator, true); + } = + @SafeVarargs public final void applyProjectDistinct(Lis= t>=2E=2E=2E auxiliaryExpressions) = throws AlgebricksException { diff --git a/asterixdb/asterix-runtime/src= /main/java/org/apache/asterix/runtime/operators/LSMSecondaryInsertDeleteWit= hNestedPlanOperatorNodePushable=2Ejava b/asterixdb/asterix-runtime/src/main= /java/org/apache/asterix/runtime/operators/LSMSecondaryInsertDeleteWithNest= edPlanOperatorNodePushable=2Ejava index 0b36774=2E=2E99b9a08 100644 --- a/a= sterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators= /LSMSecondaryInsertDeleteWithNestedPlanOperatorNodePushable=2Ejava +++ b/as= terixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/= LSMSecondaryInsertDeleteWithNestedPlanOperatorNodePushable=2Ejava @@ -31,13= +31,13 @@ import org=2Eapache=2Ehyracks=2Eapi=2Econtext=2EIHyracksTaskCon= text; import org=2Eapache=2Ehyracks=2Eapi=2Edataflow=2Evalue=2ERecordDescr= iptor; import org=2Eapache=2Ehyracks=2Eapi=2Eexceptions=2EHyracksDataExcep= tion; -import org=2Eapache=2Ehyracks=2Edataflow=2Ecommon=2Ecomm=2Eio=2EArra= yTupleBuilder; -import org=2Eapache=2Ehyracks=2Edataflow=2Ecommon=2Ecomm=2E= io=2EArrayTupleReference; import org=2Eapache=2Ehyracks=2Edataflow=2Ecommo= n=2Ecomm=2Eio=2EFrameTupleAccessor; import org=2Eapache=2Ehyracks=2Edatafl= ow=2Ecommon=2Ecomm=2Eutil=2EFrameUtils; +import org=2Eapache=2Ehyracks=2Eda= taflow=2Ecommon=2Edata=2Eaccessors=2EFrameTupleReference; import org=2Eapa= che=2Ehyracks=2Estorage=2Eam=2Ecommon=2Eapi=2EIModificationOperationCallbac= kFactory; import org=2Eapache=2Ehyracks=2Estorage=2Eam=2Ecommon=2Edataflow= =2EIIndexDataflowHelperFactory; import org=2Eapache=2Ehyracks=2Estorage=2E= am=2Ecommon=2Eophelpers=2EIndexOperation; +import org=2Eapache=2Ehyracks=2E= storage=2Eam=2Ecommon=2Etuples=2EConcatenatingTupleReference; import org= =2Eapache=2Ehyracks=2Estorage=2Eam=2Elsm=2Ecommon=2Eapi=2EILSMIndexAccessor= ; import org=2Eapache=2Ehyracks=2Estorage=2Eam=2Elsm=2Ecommon=2Edataflow= =2ELSMIndexInsertUpdateDeleteOperatorNodePushable; @@ -122,10 +122,8 @@ = private class IndexTupleInsertDelete implements IFrameWriter { = private final RecordDescriptor inputRecordDescriptor; private Fra= meTupleAccessor endOfPipelineTupleAccessor; - - // We are not writin= g the resulting tuple to a frame, we must store the result in an intermedia= te=2E - private ArrayTupleBuilder arrayTupleBuilder; - privat= e ArrayTupleReference arrayTupleReference; + private FrameTupleRefer= ence endOfPipelineTupleReference; + private ConcatenatingTupleRefere= nce endTupleReference; private IndexTupleInsertDelete(RecordDesc= riptor recordDescriptor) { this=2EinputRecordDescriptor =3D re= cordDescriptor; @@ -133,11 +131,11 @@ @Override public = void open() throws HyracksDataException { - int numSecondaryKeys= =3D inputRecordDescriptor=2EgetFieldCount(); + int indexEntryLe= ngth =3D inputRecordDescriptor=2EgetFieldCount() + numberOfPrimaryKeyAndFil= terFields; + endTupleReference =3D new ConcatenatingTupleReferen= ce(indexEntryLength); endOfPipelineTupleAccessor =3D new Fra= meTupleAccessor(inputRecordDescriptor); - arrayTupleBuilder =3D = new ArrayTupleBuilder(numberOfPrimaryKeyAndFilterFields + numSecondaryKeys)= ; - arrayTupleReference =3D new ArrayTupleReference(); + = endOfPipelineTupleReference =3D new FrameTupleReference(); } = @Override @@ -147,25 +145,35 @@ endOfPipelineTupleAc= cessor=2Ereset(buffer); int nTuple =3D endOfPipelineTupleAcces= sor=2EgetTupleCount(); for (int t =3D 0; t < nTuple; t++) { + = endOfPipelineTupleReference=2Ereset(endOfPipelineTupleAccess= or, t); + endTupleReference=2Ereset(); - //= First, add the secondary keys=2E - arrayTupleBuilder=2Erese= t(); - int nFields =3D endOfPipelineTupleAccessor=2EgetField= Count(); - for (int f =3D 0; f < nFields; f++) { - = arrayTupleBuilder=2EaddField(endOfPipelineTupleAccessor, t, f); -= } + // Add the secondary keys=2E + = endTupleReference=2EaddTuple(endOfPipelineTupleReference); - = // Next, add the primary keys and filter fields=2E - = for (int f =3D 0; f < numberOfPrimaryKeyAndFilterFields; f++) { - = arrayTupleBuilder=2EaddField(tuple=2EgetFieldData(f), tuple=2Eg= etFieldStart(f), tuple=2EgetFieldLength(f)); - } + = // Add the primary keys and filter fields=2E + endTupl= eReference=2EaddTuple(tuple); - // Finally, pass the tuple= to our accessor=2E There are only two operations: insert or delete=2E - = arrayTupleReference=2Ereset(arrayTupleBuilder=2EgetFieldEndOff= sets(), arrayTupleBuilder=2EgetByteArray()); + // Pass the t= uple to our accessor=2E There are only two operations: insert or delete=2E = if (op=2Eequals(IndexOperation=2EINSERT)) { - = workingLSMAccessor=2EforceInsert(arrayTupleReference); + = try { + workingLSMAccessor=2EforceInsert(end= TupleReference); + } catch (HyracksDataException e) { + = if (!e=2Ematches(org=2Eapache=2Ehyracks=2Eapi=2Eexce= ptions=2EErrorCode=2EDUPLICATE_KEY)) { + throw e= ; + } + } +// Showing that I am P= hD quality, changing directions for PhD to graph database, number of minute= s per slide +// + } else { - workingLSM= Accessor=2EforceDelete(arrayTupleReference); + try { + = workingLSMAccessor=2EforceDelete(endTupleReference); = + } catch (HyracksDataException e) { + = if (!e=2Ematches(org=2Eapache=2Ehyracks=2Eapi=2Eexceptions=2EErrorCod= e=2EUPDATE_OR_DELETE_NON_EXISTENT_KEY)) { + thro= w e; + } + } } = } } diff --git a/asterixdb/asterix-runtime/src/main/ja= va/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOpe= ratorNodePushable=2Ejava b/asterixdb/asterix-runtime/src/main/java/org/apac= he/asterix/runtime/operators/LSMSecondaryUpsertWithNestedPlanOperatorNodePu= shable=2Ejava index f1af496=2E=2E91f3920 100644 --- a/asterixdb/asterix-run= time/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertW= ithNestedPlanOperatorNodePushable=2Ejava +++ b/asterixdb/asterix-runtime/sr= c/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsertWithNest= edPlanOperatorNodePushable=2Ejava @@ -33,13 +33,12 @@ import org=2Eapache= =2Ehyracks=2Eapi=2Econtext=2EIHyracksTaskContext; import org=2Eapache=2Ehy= racks=2Eapi=2Edataflow=2Evalue=2ERecordDescriptor; import org=2Eapache=2Eh= yracks=2Eapi=2Eexceptions=2EHyracksDataException; -import org=2Eapache=2Ehy= racks=2Edataflow=2Ecommon=2Ecomm=2Eio=2EArrayTupleBuilder; -import org=2Eap= ache=2Ehyracks=2Edataflow=2Ecommon=2Ecomm=2Eio=2EArrayTupleReference; impo= rt org=2Eapache=2Ehyracks=2Edataflow=2Ecommon=2Ecomm=2Eio=2EFrameTupleAcces= sor; import org=2Eapache=2Ehyracks=2Edataflow=2Ecommon=2Ecomm=2Eutil=2EFra= meUtils; import org=2Eapache=2Ehyracks=2Edataflow=2Ecommon=2Edata=2Eaccess= ors=2EFrameTupleReference; import org=2Eapache=2Ehyracks=2Estorage=2Eam=2E= common=2Eapi=2EIModificationOperationCallbackFactory; import org=2Eapache= =2Ehyracks=2Estorage=2Eam=2Ecommon=2Edataflow=2EIIndexDataflowHelperFactory= ; +import org=2Eapache=2Ehyracks=2Estorage=2Eam=2Ecommon=2Etuples=2EConcate= natingTupleReference; import org=2Eapache=2Ehyracks=2Estorage=2Eam=2Elsm= =2Ecommon=2Eapi=2EILSMIndexAccessor; public class LSMSecondaryUpsertWith= NestedPlanOperatorNodePushable extends LSMSecondaryUpsertOperatorNodePushab= le { @@ -154,10 +153,7 @@ private FrameTupleAccessor endOfPipeli= neTupleAccessor; private FrameTupleReference endOfPipelineTupleRef= erence; - - // We are not writing the resulting tuple to a frame, we= must store the result in an intermediate=2E - private ArrayTupleBui= lder arrayTupleBuilder; - private ArrayTupleReference arrayTupleRefe= rence; + private ConcatenatingTupleReference endTupleReference; = private IndexTupleUnconditionalOperation(RecordDescriptor recordDesc= riptor, boolean isInsert) { this=2EinputRecordDescriptor =3D r= ecordDescriptor; @@ -166,11 +162,11 @@ @Override public= void open() throws HyracksDataException { - int numSecondaryKey= s =3D inputRecordDescriptor=2EgetFieldCount(); + int indexEntryL= ength =3D inputRecordDescriptor=2EgetFieldCount() + numberOfPrimaryKeyAndFi= lterFields; + endTupleReference =3D new ConcatenatingTupleRefere= nce(indexEntryLength); + endOfPipelineTupleAccessor =3D new Fr= ameTupleAccessor(inputRecordDescriptor); endOfPipelineTupleRef= erence =3D new FrameTupleReference(); - arrayTupleBuilder =3D ne= w ArrayTupleBuilder(numberOfPrimaryKeyAndFilterFields + numSecondaryKeys); = - arrayTupleReference =3D new ArrayTupleReference(); } = @Override @@ -180,33 +176,39 @@ endOfPipelineTupleA= ccessor=2Ereset(buffer); int nTuple =3D endOfPipelineTupleAcce= ssor=2EgetTupleCount(); for (int t =3D 0; t < nTuple; t++) { -= endOfPipelineTupleReference=2Ereset(endOfPipelineTupleAcc= essor, t); + + // Do not perform operations w/ null or missi= ng values (same behavior as atomic upserts)=2E if (hasNull= OrMissing(endOfPipelineTupleReference)) { - // Do not pe= rform operations w/ null or missing values (same behavior as atomic upserts= )=2E continue; } - /= / First, add the secondary keys=2E - arrayTupleBuilder=2Eres= et(); - int nFields =3D endOfPipelineTupleAccessor=2EgetFiel= dCount(); - for (int f =3D 0; f < nFields; f++) { - = arrayTupleBuilder=2EaddField(endOfPipelineTupleAccessor, t, f); = - } + // Add the secondary keys=2E + = endTupleReference=2Ereset(); + endTupleReference=2Ead= dTuple(endOfPipelineTupleReference); - // Next, add the pr= imary keys and filter fields=2E - for (int f =3D 0; f < numb= erOfPrimaryKeyAndFilterFields; f++) { - arrayTupleBuilde= r=2EaddField(tuple=2EgetFieldData(f), tuple=2EgetFieldStart(f), tuple=2Eget= FieldLength(f)); - } + // Add the primary key= s and filter fields=2E + endTupleReference=2EaddTuple(tuple)= ; // Finally, pass the tuple to our accessor=2E There ar= e only two operations: insert or delete=2E - arrayTupleRefer= ence=2Ereset(arrayTupleBuilder=2EgetFieldEndOffsets(), arrayTupleBuilder=2E= getByteArray()); if (this=2EisInsert) { = abstractModCallback=2EsetOp(AbstractIndexModificationOperationCallback= =2EOperation=2EINSERT); - workingLSMAccessor=2EforceInse= rt(arrayTupleReference); + try { + = workingLSMAccessor=2EforceInsert(endTupleReference); + = } catch (HyracksDataException e) { + if (!e=2Ematch= es(org=2Eapache=2Ehyracks=2Eapi=2Eexceptions=2EErrorCode=2EDUPLICATE_KEY)) = { + throw e; + } + = } } else { abstractModCa= llback=2EsetOp(AbstractIndexModificationOperationCallback=2EOperation=2EDEL= ETE); - workingLSMAccessor=2EforceDelete(arrayTupleRefer= ence); + try { + workingLSMAccess= or=2EforceDelete(endTupleReference); + } catch (HyracksD= ataException e) { + if (!e=2Ematches(org=2Eapache=2E= hyracks=2Eapi=2Eexceptions=2EErrorCode=2EUPDATE_OR_DELETE_NON_EXISTENT_KEY)= ) { + throw e; + } + = } } } } -- To view,= visit https://asterix-gerrit=2Eics=2Euci=2Eedu/c/asterixdb/+/11723 To unsu= bscribe, or for help writing mail filters, visit https://asterix-gerrit=2Ei= cs=2Euci=2Eedu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Ge= rrit-Change-Id: I328bf4e80037a23b6d44139548436f6c4f232bba Gerrit-Change-Num= ber: 11723 Gerrit-PatchSet: 1 Gerrit-Owner: Glenn Galvizo Gerrit-MessageType: newchange --OsH6RUtJxCQ= Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Glenn Galvizo has uploaded this change for review=2E

View Change

PLEASE EDIT to provide a meaningful commit message!
The following commits from your working branch will be included:

c=
ommit 8cb9a782df3ec1c8e42a59341d7d7b37d0eb5906
Author: ggalvizo <ggal=
vizo@uci=2Eedu>
Date:   Sat May 29 19:43:34 2021 -0700

    Cur=
rently using the try-catch pattern to implement the implicit DISTINCT for a=
rray index maintenance=2E

Change-Id: I328bf4e80037a23b6d44139548436f=
6c4f232bba
---
M asterixdb/asterix-algebra/src/main/java/org/apache/a=
sterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule=2Ejava
M =
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operator=
s/LSMSecondaryInsertDeleteWithNestedPlanOperatorNodePushable=2Ejava
M as=
terixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/=
LSMSecondaryUpsertWithNestedPlanOperatorNodePushable=2Ejava
3 files chan=
ged, 66 insertions(+), 49 deletions(-)

git pull ssh://asterix-ger=
rit=2Eics=2Euci=2Eedu:29418/asterixdb refs/changes/23/11723/1
diff =
--git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimize=
r/rules/IntroduceSecondaryIndexInsertDeleteRule=2Ejava b/asterixdb/asterix-=
algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondary=
IndexInsertDeleteRule=2Ejava
index c3859be=2E=2E182687d 100=
644
--- a/asterixdb/asterix-algebra/src/main/java/org/apach=
e/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule=2Ejava
+++ b/asterixdb/asterix-algebra/src/main/java/org/apache/aste=
rix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule=2Ejava@@ -449,7 +449,7 @@
                         contex=
t=2EcomputeAndSetTypeEnvironmentForOperator(unnestSourceOp);
                         UnnestBranchCreator unnestSIDXBranch =3D buildUn=
nestBranch(unnestSourceOp, index, newRecordVar,
           =
                      newMetaVar, recType, metaType, dataset=2EhasMetaPart(=
));
-                  =
      unnestSIDXBranch=2EapplyProjectDistinct();
+                        unnestSIDXBranch=2Eapp=
lyProjectOnly();
 
                        =
 // If there exists a filter expression, add it to the top of our nested pl=
an=2E
                         filterExpression =3D (primar=
yIndexModificationOp=2EgetOperation() =3D=3D Kind=2EUPSERT) ? null@@ -477,7 +477,7 @@
                             Un=
nestBranchCreator unnestBeforeSIDXBranch =3D buildUnnestBranch(unnestBefore=
SourceOp, index,
                                     prima=
ryIndexModificationOp=2EgetBeforeOpRecordVar(), beforeOpMetaVar, recType,
                                     metaType, dataset=2Ehas=
MetaPart());
-         =
                   unnestBeforeSIDXBranch=2EapplyProjectDistinct();<=
br>+                           =
 unnestBeforeSIDXBranch=2EapplyProjectOnly();
             =
                indexUpdate=2EgetNestedPlans()=2Eadd(unnestBeforeSIDXBranch=
=2EbuildBranch());
                         }
                     } else if (index=2EgetIndexType() =3D=3D IndexType=
=2EARRAY && isBulkload) {
@@ -1010,6 +1010,13 @@
             return varRef;
         }=

 
+        =
public final void applyProjectOnly() throws AlgebricksException {+            List<LogicalVa=
riable> projectVars =3D new ArrayList<>(this=2ElastFieldVars);
+            ProjectOper=
ator projectOperator =3D new ProjectOperator(projectVars);
+            projectOperator=2EsetSou=
rceLocation(sourceLoc);
+            this=2EcurrentTop =3D introduceNewOp(currentTop, projectOpe=
rator, true);
+      =
  }
+
         @SafeVarargs
         public final void applyProj=
ectDistinct(List<Mutable<ILogicalExpression>>=2E=2E=2E auxiliar=
yExpressions)
                 throws AlgebricksException {=

diff --git a/asterixdb/asterix-runtime/src/main/java/org/a=
pache/asterix/runtime/operators/LSMSecondaryInsertDeleteWithNestedPlanOpera=
torNodePushable=2Ejava b/asterixdb/asterix-runtime/src/main/java/org/apache=
/asterix/runtime/operators/LSMSecondaryInsertDeleteWithNestedPlanOperatorNo=
dePushable=2Ejava
index 0b36774=2E=2E99b9a08 100644<=
br>--- a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/r=
untime/operators/LSMSecondaryInsertDeleteWithNestedPlanOperatorNodePushable=
=2Ejava
+++ b/asterixdb/asterix-runtime/src/main/java/org/a=
pache/asterix/runtime/operators/LSMSecondaryInsertDeleteWithNestedPlanOpera=
torNodePushable=2Ejava
@@ -31,13 +31,13 @@
=
 import org=2Eapache=2Ehyracks=2Eapi=2Econtext=2EIHyracksTaskContext;
 import org=2Eapache=2Ehyracks=2Eapi=2Edataflow=2Evalue=2ERecord=
Descriptor;
 import org=2Eapache=2Ehyracks=2Eapi=2Eexceptio=
ns=2EHyracksDataException;
-import org=2Eapache=2Ehyracks=2Edataflow=2Ecommon=2Ecomm=2Eio=2EArrayT=
upleBuilder;
-import or=
g=2Eapache=2Ehyracks=2Edataflow=2Ecommon=2Ecomm=2Eio=2EArrayTupleReference;=

 import org=2Eapache=2Ehyracks=2Edataflow=2Ecommon=2Ecomm=
=2Eio=2EFrameTupleAccessor;
 import org=2Eapache=2Ehyracks=
=2Edataflow=2Ecommon=2Ecomm=2Eutil=2EFrameUtils;
+import org=2Eapache=2Ehyracks=2Edataflow=2Ecom=
mon=2Edata=2Eaccessors=2EFrameTupleReference;
 import org=
=2Eapache=2Ehyracks=2Estorage=2Eam=2Ecommon=2Eapi=2EIModificationOperationC=
allbackFactory;
 import org=2Eapache=2Ehyracks=2Estorage=2E=
am=2Ecommon=2Edataflow=2EIIndexDataflowHelperFactory;
 impo=
rt org=2Eapache=2Ehyracks=2Estorage=2Eam=2Ecommon=2Eophelpers=2EIndexOperat=
ion;
+import org=2Eap=
ache=2Ehyracks=2Estorage=2Eam=2Ecommon=2Etuples=2EConcatenatingTupleReferen=
ce;
 import org=2Eapache=2Ehyracks=2Estorage=2Eam=2Elsm=2Ec=
ommon=2Eapi=2EILSMIndexAccessor;
 import org=2Eapache=2Ehyr=
acks=2Estorage=2Eam=2Elsm=2Ecommon=2Edataflow=2ELSMIndexInsertUpdateDeleteO=
peratorNodePushable;
 
@@ -122,10 +122,8 @@=

     private class IndexTupleInsertDelete implements IFram=
eWriter {
         private final RecordDescriptor inputReco=
rdDescriptor;
         private FrameTupleAccessor endOfPipe=
lineTupleAccessor;
-
-        // We are not wr=
iting the resulting tuple to a frame, we must store the result in an interm=
ediate=2E
-        priv=
ate ArrayTupleBuilder arrayTupleBuilder;
-        private ArrayTupleReference arrayTupleReference;=

+        private Fra=
meTupleReference endOfPipelineTupleReference;
+        private ConcatenatingTupleReference endTu=
pleReference;
 
         private IndexTuple=
InsertDelete(RecordDescriptor recordDescriptor) {
         =
    this=2EinputRecordDescriptor =3D recordDescriptor;
@@ -=
133,11 +131,11 @@
 
         @Override
         public void open() throws HyracksDataException {
-            int numSecondar=
yKeys =3D inputRecordDescriptor=2EgetFieldCount();
+            int indexEntryLength =3D inputRe=
cordDescriptor=2EgetFieldCount() + numberOfPrimaryKeyAndFilterFields;
+            endTupleRefer=
ence =3D new ConcatenatingTupleReference(indexEntryLength);
 
             endOfPipelineTupleAccessor =3D new FrameTup=
leAccessor(inputRecordDescriptor);
-            arrayTupleBuilder =3D new ArrayTupleBuilder(number=
OfPrimaryKeyAndFilterFields + numSecondaryKeys);
-            arrayTupleReference =3D new ArrayTup=
leReference();
+     =
       endOfPipelineTupleReference =3D new FrameTupleReference();         }
 
         @Override
@@ -147,25 +145,35 @@
             endOfPipelin=
eTupleAccessor=2Ereset(buffer);
             int nTuple =3D=
 endOfPipelineTupleAccessor=2EgetTupleCount();
            =
 for (int t =3D 0; t < nTuple; t++) {
+                endOfPipelineTupleReference=2Ereset(en=
dOfPipelineTupleAccessor, t);
+                endTupleReference=2Ereset();
 
-                // First=
, add the secondary keys=2E
-                arrayTupleBuilder=2Ereset();
-                int nFields =3D endOfPipelineT=
upleAccessor=2EgetFieldCount();
-                for (int f =3D 0; f < nFields; f++) {-                    arrayTuple=
Builder=2EaddField(endOfPipelineTupleAccessor, t, f);
-                }
+                // Add the secondary keys=2E<=
/span>
+                endT=
upleReference=2EaddTuple(endOfPipelineTupleReference);
 
-                // Next,=
 add the primary keys and filter fields=2E
-                for (int f =3D 0; f < numberOfPrima=
ryKeyAndFilterFields; f++) {
-                    arrayTupleBuilder=2EaddField(tuple=2EgetFieldDat=
a(f), tuple=2EgetFieldStart(f), tuple=2EgetFieldLength(f));
-                }
+                // Add the primary keys=
 and filter fields=2E
+                endTupleReference=2EaddTuple(tuple);
 
-                // Final=
ly, pass the tuple to our accessor=2E There are only two operations: insert=
 or delete=2E
-        =
        arrayTupleReference=2Ereset(arrayTupleBuilder=2EgetFieldEndOffsets(=
), arrayTupleBuilder=2EgetByteArray());
+                // Pass the tuple to our accessor=2E Th=
ere are only two operations: insert or delete=2E
          =
       if (op=2Eequals(IndexOperation=2EINSERT)) {
-                    workingLSMAccessor=2Eforce=
Insert(arrayTupleReference);
+                    try {
+                        workingLSMAccessor=2EforceInsert(end=
TupleReference);
+   =
                 } catch (HyracksDataException e) {
+                        if (!e=2Ematches(=
org=2Eapache=2Ehyracks=2Eapi=2Eexceptions=2EErrorCode=2EDUPLICATE_KEY)) {
+                     =
       throw e;
+    =
                    }
+                    }
+// Showing that I am PhD quality, changing directions for PhD to graph =
database, number of minutes per slide
+//
=
+
                 } else {
-                    workingLSMAccessor=2EforceDelete(=
arrayTupleReference);
+                    try {
+                        workingLSMAccessor=2EforceDelete(endTupleRe=
ference);
+          =
          } catch (HyracksDataException e) {
+                        if (!e=2Ematches(org=2Eapa=
che=2Ehyracks=2Eapi=2Eexceptions=2EErrorCode=2EUPDATE_OR_DELETE_NON_EXISTEN=
T_KEY)) {
+          =
                  throw e;
+                        }
+                    }
                 }
             }
         }
diff -=
-git a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/o=
perators/LSMSecondaryUpsertWithNestedPlanOperatorNodePushable=2Ejava b/aste=
rixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LS=
MSecondaryUpsertWithNestedPlanOperatorNodePushable=2Ejava
i=
ndex f1af496=2E=2E91f3920 100644
--- a/asterixdb/asterix-ru=
ntime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryUpsert=
WithNestedPlanOperatorNodePushable=2Ejava
+++ b/asterixdb/a=
sterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecond=
aryUpsertWithNestedPlanOperatorNodePushable=2Ejava
@@ -33,1=
3 +33,12 @@
 import org=2Eapache=2Ehyracks=2Eapi=2Econtext=
=2EIHyracksTaskContext;
 import org=2Eapache=2Ehyracks=2Eap=
i=2Edataflow=2Evalue=2ERecordDescriptor;
 import org=2Eapac=
he=2Ehyracks=2Eapi=2Eexceptions=2EHyracksDataException;
-import org=2Eapache=2Ehyracks=2Edataflow=
=2Ecommon=2Ecomm=2Eio=2EArrayTupleBuilder;
-import org=2Eapache=2Ehyracks=2Edataflow=2Ecommon=2Eco=
mm=2Eio=2EArrayTupleReference;
 import org=2Eapache=2Ehyrac=
ks=2Edataflow=2Ecommon=2Ecomm=2Eio=2EFrameTupleAccessor;
 i=
mport org=2Eapache=2Ehyracks=2Edataflow=2Ecommon=2Ecomm=2Eutil=2EFrameUtils=
;
 import org=2Eapache=2Ehyracks=2Edataflow=2Ecommon=2Edata=
=2Eaccessors=2EFrameTupleReference;
 import org=2Eapache=2E=
hyracks=2Estorage=2Eam=2Ecommon=2Eapi=2EIModificationOperationCallbackFacto=
ry;
 import org=2Eapache=2Ehyracks=2Estorage=2Eam=2Ecommon=
=2Edataflow=2EIIndexDataflowHelperFactory;
+import org=2Eapache=2Ehyracks=2Estorage=2Eam=2Ecommo=
n=2Etuples=2EConcatenatingTupleReference;
 import org=2Eapa=
che=2Ehyracks=2Estorage=2Eam=2Elsm=2Ecommon=2Eapi=2EILSMIndexAccessor;
 
 public class LSMSecondaryUpsertWithNestedPla=
nOperatorNodePushable extends LSMSecondaryUpsertOperatorNodePushable {
@@ -154,10 +153,7 @@
 
        =
 private FrameTupleAccessor endOfPipelineTupleAccessor;
   =
      private FrameTupleReference endOfPipelineTupleReference;
-
-        // We are not writing the resulting tuple to a =
frame, we must store the result in an intermediate=2E
-        private ArrayTupleBuilder arrayTupl=
eBuilder;
-        priv=
ate ArrayTupleReference arrayTupleReference;
+        private ConcatenatingTupleReference endTup=
leReference;
 
         private IndexTupleU=
nconditionalOperation(RecordDescriptor recordDescriptor, boolean isInsert) =
{
             this=2EinputRecordDescriptor =3D recordDescr=
iptor;
@@ -166,11 +162,11 @@
 
         @Override
         public void open() throws H=
yracksDataException {
-=
            int numSecondaryKeys =3D inputRecordDescriptor=2EgetFieldCount(=
);
+            int i=
ndexEntryLength =3D inputRecordDescriptor=2EgetFieldCount() + numberOfPrima=
ryKeyAndFilterFields;
+            endTupleReference =3D new ConcatenatingTupleReference(indexEn=
tryLength);
+<=
br>             endOfPipelineTupleAccessor =3D new FrameTupleAccessor=
(inputRecordDescriptor);
             endOfPipelineTupleRef=
erence =3D new FrameTupleReference();
-            arrayTupleBuilder =3D new ArrayTupleBuilder(num=
berOfPrimaryKeyAndFilterFields + numSecondaryKeys);
-            arrayTupleReference =3D new Arr=
ayTupleReference();
         }
 
=
         @Override
@@ -180,33 +176,39 @@
             endOfPipelineTupleAccessor=2Ereset(buffer);
             int nTuple =3D endOfPipelineTupleAccessor=2EgetTupleCount();=

             for (int t =3D 0; t < nTuple; t++) {
-
          =
       endOfPipelineTupleReference=2Ereset(endOfPipelineTupleAccessor, t);<=
/span>
+
+                // Do not perform opera=
tions w/ null or missing values (same behavior as atomic upserts)=2E=

                 if (hasNullOrMissing(endOfPipelineTupleReference=
)) {
-                 =
   // Do not perform operations w/ null or missing values (same behavior as=
 atomic upserts)=2E
                     continue;                 }
 
-                // First, add the secondary keys=2E<=
/span>
-                arrayT=
upleBuilder=2Ereset();
=
-                int nFields =3D endOfPipelineTupleAccessor=2EgetFieldCount=
();
-                fo=
r (int f =3D 0; f < nFields; f++) {
-                    arrayTupleBuilder=2EaddField(endOfPipe=
lineTupleAccessor, t, f);
-                }
+                // Add the secondary keys=2E
+                endTupleReference=2Ereset();
+                endTupl=
eReference=2EaddTuple(endOfPipelineTupleReference);
 
-                // Next, ad=
d the primary keys and filter fields=2E
-                for (int f =3D 0; f < numberOfPrimaryK=
eyAndFilterFields; f++) {
-                    arrayTupleBuilder=2EaddField(tuple=2EgetFieldData(f=
), tuple=2EgetFieldStart(f), tuple=2EgetFieldLength(f));
-                }
+                // Add the primary keys a=
nd filter fields=2E
+=
                endTupleReference=2EaddTuple(tuple);
 
                 // Finally, pass the tuple to our accessor=2E =
There are only two operations: insert or delete=2E
-                arrayTupleReference=2Ereset(ar=
rayTupleBuilder=2EgetFieldEndOffsets(), arrayTupleBuilder=2EgetByteArray())=
;
                 if (this=2EisInsert) {
 =
                    abstractModCallback=2EsetOp(AbstractIndexModificationOp=
erationCallback=2EOperation=2EINSERT);
-                    workingLSMAccessor=2EforceInsert(array=
TupleReference);
+   =
                 try {
+               =
     } catch (HyracksDataException e) {
+                        if (!e=2Ematches(org=2Eapache=
=2Ehyracks=2Eapi=2Eexceptions=2EErrorCode=2EDUPLICATE_KEY)) {
+                            throw=
 e;
+                =
        }
+          =
          }
                 } else {
     =
                abstractModCallback=2EsetOp(AbstractIndexModificationOperat=
ionCallback=2EOperation=2EDELETE);
-                    workingLSMAccessor=2EforceDelete(arrayTupl=
eReference);
+       =
             try {
+ =
                       workingLSMAccessor=2EforceDelete(endTupleReference);=

+                   =
 } catch (HyracksDataException e) {
+                        if (!e=2Ematches(org=2Eapache=2Ehyr=
acks=2Eapi=2Eexceptions=2EErrorCode=2EUPDATE_OR_DELETE_NON_EXISTENT_KEY)) {=

+                   =
         throw e;
+  =
                      }
+                    }
                 }
             }
         }

To view, visit change 11723=2E To unsubscribe, or for help writing m=
ail filters, visit settings=2E



 Gerrit-Project: asterixdb 


 Ger=
rit-Branch: master 


 Gerrit-Change-Id: I3=
28bf4e80037a23b6d44139548436f6c4f232bba 


=
 Gerrit-Change-Number: 11723 


 Gerrit-Pat=
chSet: 1 


 Gerrit-Owner: Glenn Galvizo &l=
t;ggalvizo@uci=2Eedu> 


 Gerrit-Message=
Type: newchange 



--OsH6RUtJxCQ=--