From notifications-return-158412-archive-asf-public=cust-asf.ponee.io@asterixdb.apache.org Fri Jun 25 17:37:18 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 09773180181 for ; Fri, 25 Jun 2021 19:37:18 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id 419E03F2B8 for ; Fri, 25 Jun 2021 17:37:17 +0000 (UTC) Received: (qmail 87448 invoked by uid 500); 25 Jun 2021 17:37:17 -0000 Mailing-List: contact notifications-help@asterixdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.apache.org Delivered-To: mailing list notifications@asterixdb.apache.org Received: (qmail 87364 invoked by uid 99); 25 Jun 2021 17:37:16 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2021 17:37:16 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id 8519CC03D8 for ; Fri, 25 Jun 2021 17:37:15 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: 0.213 X-Spam-Level: X-Spam-Status: No, score=0.213 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=0.2, SPF_NONE=0.001, T_KAM_HTML_FONT_INVALID=0.01, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id 9Sxq_2kD1x35 for ; Fri, 25 Jun 2021 17:37:11 +0000 (UTC) Received-SPF: None (mailfrom) identity=mailfrom; client-ip=128.195.1.133; helo=adrian-monk-v3.ics.uci.edu; envelope-from=do-not-reply@asterix-gerrit.ics.uci.edu; receiver= Received: from adrian-monk-v3.ics.uci.edu (adrian-monk-v3.ics.uci.edu [128.195.1.133]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 1D6B17FE60 for ; Fri, 25 Jun 2021 17:37:09 +0000 (UTC) Received: from f65bc2393bbf (vitalstatistix.ics.uci.edu [128.195.52.38]) by adrian-monk-v3.ics.uci.edu (Postfix) with ESMTP id 48F26C02E2B9; Fri, 25 Jun 2021 10:37:01 -0700 (PDT) X-Gerrit-PatchSet: 1 Date: Fri, 25 Jun 2021 17:37:01 +0000 From: AsterixDB Code Review To: Till Westmann Message-ID: Auto-Submitted: auto-generated X-Gerrit-MessageType: newchange Subject: Change in asterixdb[master]: [NO ISSUE][IDX] Refactoring of array index code. X-Gerrit-Change-Id: Id337d1032796e1d1c2ce68ea2861b4a81dd19aa5 X-Gerrit-Change-Number: 12063 X-Gerrit-Project: asterixdb X-Gerrit-ChangeURL: X-Gerrit-Commit: 2810d55eb48a4a2647f8372ddcdd0168967c4308 References: Reply-To: ggalvizo@uci.edu, dmitry.lychagin@couchbase.com, notifications@asterixdb.apache.org, lwhaywhu@gmail.com, tillw@apache.org MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/3.0.12 Content-Type: multipart/alternative; boundary="/daqNoEYv34="; charset=UTF-8 X-ICS-MailScanner-Information: Please send mail to helpdesk@ics.uci.edu or more information X-ICS-MailScanner-ID: 48F26C02E2B9.A6683 X-ICS-MailScanner: Found to be clean X-ICS-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-0.985, required 5, ALL_TRUSTED -1.00, FSL_HELO_NON_FQDN_1 0.00, HELO_NO_DOMAIN 0.00, HTML_MESSAGE 0.00, T_KAM_HTML_FONT_INVALID 0.01, URIBL_BLOCKED 0.00, WEIRD_PORT 0.00) X-ICS-MailScanner-From: do-not-reply@asterix-gerrit.ics.uci.edu --/daqNoEYv34= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable From Glenn Galvizo : Glenn Galvizo has uploaded this c= hange for review=2E ( https://asterix-gerrit=2Eics=2Euci=2Eedu/c/asterixdb/= +/12063 ) Change subject: [NO ISSUE][IDX] Refactoring of array index code= =2E =2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E= =2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E= =2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E [NO ISS= UE][IDX] Refactoring of array index code=2E - user model changes: no - sto= rage format changes: no - interface changes: no Refactor: no longer using = depth indicators, rather using UNNEST flags with flattened field names=2E W= e can no longer have back-to-back UNNESTs (i=2Ee=2E arrays within arrays wi= thout an ASSIGN intermediate), as no such array index can be created=2E=2E = Change-Id: Id337d1032796e1d1c2ce68ea2861b4a81dd19aa5 --- M asterixdb/aster= ix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecond= aryIndexInsertDeleteRule=2Ejava M asterixdb/asterix-algebra/src/main/java/o= rg/apache/asterix/optimizer/rules/am/AccessMethodUtils=2Ejava M asterixdb/a= sterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/ArrayBTr= eeAccessMethod=2Ejava M asterixdb/asterix-metadata/src/main/java/org/apache= /asterix/metadata/declared/ArrayBTreeResourceFactoryProvider=2Ejava M aster= ixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/Array= IndexUtil=2Ejava M asterixdb/asterix-metadata/src/main/java/org/apache/aste= rix/metadata/utils/KeyFieldTypeUtil=2Ejava M asterixdb/asterix-metadata/src= /main/java/org/apache/asterix/metadata/utils/SecondaryArrayIndexBTreeOperat= ionsHelper=2Ejava M asterixdb/asterix-metadata/src/main/java/org/apache/ast= erix/metadata/utils/TypeUtil=2Ejava 8 files changed, 197 insertions(+), 191= deletions(-) git pull ssh://asterix-gerrit=2Eics=2Euci=2Eedu:29418/as= terixdb refs/changes/63/12063/1 diff --git a/asterixdb/asterix-algebra/src= /main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsert= DeleteRule=2Ejava b/asterixdb/asterix-algebra/src/main/java/org/apache/aste= rix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule=2Ejava index c3= 859be=2E=2E81c6c2a 100644 --- a/asterixdb/asterix-algebra/src/main/java/org= /apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule=2Ej= ava +++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimi= zer/rules/IntroduceSecondaryIndexInsertDeleteRule=2Ejava @@ -728,10 +728,10= @@ // Walk the array path=2E List flatFirstFieldName =3D ArrayIndexUtil=2EgetFlattenedKeyFieldNames( = workingElement=2EgetUnnestList(), workingElement=2EgetPr= ojectList()=2Eget(0)); - List firstArrayIndicator = =3D ArrayIndexUtil - =2EgetArrayDepthIndicator(worki= ngElement=2EgetUnnestList(), workingElement=2EgetProjectList()=2Eget(0)); += List firstUnnestFlags =3D ArrayIndexUtil=2EgetUnne= stFlags(workingElement=2EgetUnnestList(), + workingE= lement=2EgetProjectList()=2Eget(0)); ArrayIndexUtil=2Ewalk= ArrayPath((isOpenOrNestedField) ? null : recordType, flatFirstFieldName, - = firstArrayIndicator, branchCreator); + = firstUnnestFlags, branchCreator); // For all o= ther elements in the PROJECT list, add an assign=2E for (i= nt j =3D 1; j < workingElement=2EgetProjectList()=2Esize(); j++) { @@ -1054= ,25 +1054,19 @@ @Override public void executeActionOnEa= chArrayStep(ARecordType startingStepRecordType, IAType workingType, - = List fieldName, boolean isFirstArrayStep, boolean isFirst= UnnestInStep, - boolean isLastUnnestInIntermediateStep) thro= ws AlgebricksException { + List fieldName, boolean i= sFirstArrayStep, boolean isLastUnnestInIntermediateStep) + t= hrows AlgebricksException { if (!isFirstWalk) { = // We have already built the UNNEST path, do not build again=2E = return; } + // Get the field we want to = UNNEST from our record=2E ILogicalExpression accessToUnnestVar= ; - if (isFirstUnnestInStep) { - // This is the f= irst UNNEST step=2E Get the field we want to UNNEST from our record=2E - = accessToUnnestVar =3D (startingStepRecordType !=3D null) - = ? getFieldAccessFunction(new MutableObject<>(createLast= RecordVarRef()), - startingStepRecordType=2E= getFieldIndex(fieldName=2Eget(0)), fieldName) - : ge= tFieldAccessFunction(new MutableObject<>(createLastRecordVarRef()), -1, fie= ldName); - } else { - // This is the second+ UNNE= ST step=2E Refer back to the previously unnested variable=2E - = accessToUnnestVar =3D new VariableReferenceExpression(this=2ElastFieldVa= rs=2Eget(0)); - this=2ElastFieldVars=2Eclear(); - = } + accessToUnnestVar =3D (startingStepRecordType !=3D null) + = ? getFieldAccessFunction(new MutableObject<>(createLastR= ecordVarRef()), + startingStepRecordType=2EgetFi= eldIndex(fieldName=2Eget(0)), fieldName) + : getFieldAcc= essFunction(new MutableObject<>(createLastRecordVarRef()), -1, fieldName); = UnnestingFunctionCallExpression scanCollection =3D new Unnesti= ngFunctionCallExpression( BuiltinFunctions=2EgetBuilti= nFunctionInfo(BuiltinFunctions=2ESCAN_COLLECTION), Col= lections=2EsingletonList(new MutableObject<>(accessToUnnestVar))); diff --g= it a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/r= ules/am/AccessMethodUtils=2Ejava b/asterixdb/asterix-algebra/src/main/java/= org/apache/asterix/optimizer/rules/am/AccessMethodUtils=2Ejava index c78e89= f=2E=2E3f4d3f2 100644 --- a/asterixdb/asterix-algebra/src/main/java/org/apa= che/asterix/optimizer/rules/am/AccessMethodUtils=2Ejava +++ b/asterixdb/ast= erix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/AccessMeth= odUtils=2Ejava @@ -3060,7 +3060,6 @@ MutableInt fieldSource =3D ne= w MutableInt(0); ARecordType workingRecordType =3D subTree=2EgetRe= cordType(); - // TODO: (GLENN) Refactor this to use ArrayIndexUtil= =2E // Iterate through our array index structure=2E We must match = the depth and field names for the caller's variable // to qualify = for an array-index optimization=2E LogicalVariable varFromParent = =3D assignVar; diff --git a/asterixdb/asterix-algebra/src/main/java/org/apa= che/asterix/optimizer/rules/am/ArrayBTreeAccessMethod=2Ejava b/asterixdb/as= terix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/ArrayBTre= eAccessMethod=2Ejava index 44e4a18=2E=2Ece6b407 100644 --- a/asterixdb/aste= rix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/ArrayBTreeA= ccessMethod=2Ejava +++ b/asterixdb/asterix-algebra/src/main/java/org/apache= /asterix/optimizer/rules/am/ArrayBTreeAccessMethod=2Ejava @@ -99,7 +99,6 @@= @Override protected IAType getIndexedKeyType(Index=2EIIndexDet= ails chosenIndexDetails, int keyPos) throws CompilationException { - = // TODO (GLENN): This assumes a flattened key list=2E Refactor / clarify t= his when removing depth indicators=2E Index=2EArrayIndexDetails ar= rayIndexDetails =3D (Index=2EArrayIndexDetails) chosenIndexDetails; = int elementPos =3D 0; for (Index=2EArrayIndexElement e : arrayIn= dexDetails=2EgetElementList()) { diff --git a/asterixdb/asterix-metadata/sr= c/main/java/org/apache/asterix/metadata/declared/ArrayBTreeResourceFactoryP= rovider=2Ejava b/asterixdb/asterix-metadata/src/main/java/org/apache/asteri= x/metadata/declared/ArrayBTreeResourceFactoryProvider=2Ejava index 5654d16= =2E=2Ec0bdc75 100644 --- a/asterixdb/asterix-metadata/src/main/java/org/apa= che/asterix/metadata/declared/ArrayBTreeResourceFactoryProvider=2Ejava +++ = b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/decl= ared/ArrayBTreeResourceFactoryProvider=2Ejava @@ -18,7 +18,6 @@ */ packa= ge org=2Eapache=2Easterix=2Emetadata=2Edeclared; -import java=2Eutil=2ELi= st; import java=2Eutil=2EMap; import org=2Eapache=2Easterix=2Ecommon=2E= config=2EDatasetConfig=2EDatasetType; @@ -134,10 +133,8 @@ = sourceType =3D metaType; } for (int i =3D 0; i <= e=2EgetProjectList()=2Esize(); i++) { - List projec= t =3D e=2EgetProjectList()=2Eget(i); Pair= keyTypePair =3D ArrayIndexUtil=2EgetNonNullableOpenFieldType(e=2EgetTypeLi= st()=2Eget(i), - ArrayIndexUtil=2EgetFlattenedKeyFie= ldNames(e=2EgetUnnestList(), project), sourceType, - = ArrayIndexUtil=2EgetArrayDepthIndicator(e=2EgetUnnestList(), project)); + = e=2EgetUnnestList(), e=2EgetProjectList()=2Eget(i), = sourceType); IAType keyType =3D keyTypePair=2Efirst; = secondaryTypeTraits[secondaryTypeTraitPos++] =3D typeTraitProvi= der=2EgetTypeTrait(keyType); } @@ -175,10 +172,8 @@ = sourceType =3D metaType; } for (int i =3D = 0; i < e=2EgetProjectList()=2Esize(); i++) { - List = project =3D e=2EgetProjectList()=2Eget(i); Pair keyTypePair =3D ArrayIndexUtil=2EgetNonNullableOpenFieldType(e=2Eget= TypeList()=2Eget(i), - ArrayIndexUtil=2EgetFlattened= KeyFieldNames(e=2EgetUnnestList(), project), sourceType, - = ArrayIndexUtil=2EgetArrayDepthIndicator(e=2EgetUnnestList(), project= )); + e=2EgetUnnestList(), e=2EgetProjectList()=2Ege= t(i), sourceType); IAType keyType =3D keyTypePair=2Efirst;= secondaryCmpFactories[secondaryCmpFactoriesPos++] =3D = cmpFactoryProvider=2EgetBinaryComparatorFactory(keyTy= pe, true); diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache= /asterix/metadata/utils/ArrayIndexUtil=2Ejava b/asterixdb/asterix-metadata/= src/main/java/org/apache/asterix/metadata/utils/ArrayIndexUtil=2Ejava index= f4dfe56=2E=2E0778ad9 100644 --- a/asterixdb/asterix-metadata/src/main/java= /org/apache/asterix/metadata/utils/ArrayIndexUtil=2Ejava +++ b/asterixdb/as= terix-metadata/src/main/java/org/apache/asterix/metadata/utils/ArrayIndexUt= il=2Ejava @@ -39,14 +39,18 @@ public class ArrayIndexUtil { /** - = * @deprecated Use the project + unnest scheme instead of array indicator= s=2E + * Similar function to Index's "getSubFieldType", but accounts fo= r array fields as well=2E */ - public static IAType getSubFieldInA= rrayType(ARecordType recordType, List subFieldName, - Li= st arrayDepthIndicators) throws AlgebricksException { - IAT= ype subType =3D recordType=2EgetFieldType(subFieldName=2Eget(0)); - = for (int i =3D 1; i < subFieldName=2Esize(); i++) { + public static IATy= pe getSubFieldType(ARecordType recordType, List> unnestList, += List projectList) throws AlgebricksException { + = List flattenedFieldName =3D ArrayIndexUtil=2EgetFlattenedKeyFieldN= ames(unnestList, projectList); + List unnestFlags =3D Array= IndexUtil=2EgetUnnestFlags(unnestList, projectList); + IAType subTyp= e =3D recordType=2EgetFieldType(flattenedFieldName=2Eget(0)); + + fo= r (int i =3D 1; i < flattenedFieldName=2Esize(); i++) { if (su= bType =3D=3D null) { return null; + } else if= (subType=2EgetTypeTag()=2Eequals(ATypeTag=2EUNION)) { // = Support enforced types here=2E subType =3D ((AUnionType) s= ubType)=2EgetActualType(); @@ -56,31 +60,30 @@ = "Field accessor is not defined for values of type " + subType=2EgetTypeTag= ()); } } - if (subType=2EgetTypeTa= g()=2Eequals(ATypeTag=2EOBJECT) && arrayDepthIndicators=2Eget(i - 1) =3D=3D= 0) { - subType =3D ((ARecordType) subType)=2EgetFieldType(s= ubFieldName=2Eget(i)); + + if (subType=2EgetTypeTag()=2Eequals(A= TypeTag=2EOBJECT) && !unnestFlags=2Eget(i - 1)) { + subType = =3D ((ARecordType) subType)=2EgetFieldType(flattenedFieldName=2Eget(i)); + = } else if ((subType=2EgetTypeTag()=2Eequals(ATypeTag=2EARRAY) = || subType=2EgetTypeTag()=2Eequals(ATypeTag=2EMULTISET)) - = && arrayDepthIndicators=2Eget(i - 1) > 0) { - for (int j = =3D 0; j < arrayDepthIndicators=2Eget(i - 1); j++) { - s= ubType =3D TypeComputeUtils=2EextractListItemType(subType); - = } - subType =3D (subType !=3D null) ? ((ARecordType) subTy= pe)=2EgetFieldType(subFieldName=2Eget(i)) : null; + && u= nnestFlags=2Eget(i - 1)) { + subType =3D TypeComputeUtils=2E= extractListItemType(subType); + subType =3D (subType !=3D nu= ll) ? ((ARecordType) subType)=2EgetFieldType(flattenedFieldName=2Eget(i)) := null; + } else { throw new AsterixException(= ErrorCode=2ECOMPILATION_ERROR, - (arrayDepthIndicato= rs=2Eget(i - 1) > 0) - ? "Object type given,= but array depth indicator is " + "non-zero=2E" - = : "Array/multiset type given, but array depth indicator is zero=2E");= + unnestFlags=2Eget(i - 1) ? "Object type given, bu= t unnest flag is also raised=2E" + : "Array/= multiset type given, but unnest flag is lowered=2E"); } = } - if (subType !=3D null && arrayDepthIndicators=2Eget(arrayDept= hIndicators=2Esize() - 1) > 0) { + + if (subType !=3D null && unnest= Flags=2Eget(unnestFlags=2Esize() - 1)) { // If the end field i= s an array, we must extract the list item here as well=2E - for = (int j =3D 0; j < arrayDepthIndicators=2Eget(arrayDepthIndicators=2Esize() = - 1); j++) { - if (subType instanceof AbstractCollectionType= ) { - subType =3D TypeComputeUtils=2EextractListItemType= (subType); - } else { - throw new Asterix= Exception(ErrorCode=2ECOMPILATION_ERROR, - "Arra= y type expected for last term, but given: " - = + ((subType !=3D null) ? subType=2EgetTypeTag() : "null")); - = } + if (subType instanceof AbstractCollectionType) { + = subType =3D TypeComputeUtils=2EextractListItemType(subType); = + + } else { + throw new AsterixException(ErrorCo= de=2ECOMPILATION_ERROR, + "Array type expected for l= ast term, but given: " + subType=2EgetTypeTag()); } }= return subType; @@ -88,15 +91,18 @@ /** * Given a pa= th of complex types (i=2Ee=2E lists + records), determine the nullability o= f the field=2E - * @deprecated Use the project + unnest scheme instead = of array indicators=2E */ - public static boolean isSubFieldNullab= le(ARecordType recordType, List subFieldName, - List arrayIndicators) throws AlgebricksException { - IAType subType= =3D recordType=2EgetFieldType(subFieldName=2Eget(0)); - for (int i = =3D 1; i < subFieldName=2Esize(); i++) { + public static boolean isSubFi= eldNullable(ARecordType recordType, List> unnestList, + = List projectList) throws AlgebricksException { + List flattenedFieldName =3D ArrayIndexUtil=2EgetFlattenedKeyFieldNames(un= nestList, projectList); + List unnestFlags =3D ArrayIndexUt= il=2EgetUnnestFlags(unnestList, projectList); + IAType subType =3D r= ecordType=2EgetFieldType(flattenedFieldName=2Eget(0)); + + for (int = i =3D 1; i < flattenedFieldName=2Esize(); i++) { if (subType = =3D=3D null) { return true; } + = if (subType=2EgetTypeTag()=2Eequals(ATypeTag=2EUNION)) { i= f (NonTaggedFormatUtil=2EisOptional(subType)) { return= true; @@ -109,12 +115,12 @@ } if (subType inst= anceof ARecordType) { - subType =3D ((ARecordType) subType)= =2EgetFieldType(subFieldName=2Eget(i)); - } else if (subType ins= tanceof AbstractCollectionType && arrayIndicators=2Eget(i - 1) > 0) { - = for (int j =3D 0; j < arrayIndicators=2Eget(i - 1); j++) { - = subType =3D TypeComputeUtils=2EextractListItemType(subType= ); - } - subType =3D (subType !=3D null) ? ((= ARecordType) subType)=2EgetFieldType(subFieldName=2Eget(i)) : null; + = subType =3D ((ARecordType) subType)=2EgetFieldType(flattenedField= Name=2Eget(i)); + + } else if (subType instanceof AbstractCollec= tionType && unnestFlags=2Eget(i - 1)) { + subType =3D TypeCo= mputeUtils=2EextractListItemType(subType); + subType =3D (su= bType !=3D null) ? ((ARecordType) subType)=2EgetFieldType(flattenedFieldNam= e=2Eget(i)) : null; + } else { throw Compilat= ionException=2Ecreate(ErrorCode=2ECOMPILATION_ILLEGAL_STATE, = "Illegal field type " + subType=2EgetTypeTag() + " when checking= field nullability"); @@ -125,32 +131,37 @@ /** * Similar func= tion to Index's "getNonNullableOpenFieldType", but accounts for array field= s as well=2E - * @deprecated Use the project + unnest scheme instead of= array indicators=2E */ - public static Pair getN= onNullableOpenFieldType(IAType fieldType, List fieldName, - = ARecordType recType, List arrayIndicators) throws AlgebricksEx= ception { + public static Pair getNonNullableOpenFieldT= ype(IAType fieldType, List> unnestList, + List projectList, ARecordType recType) throws AlgebricksException { = Pair keyPairType =3D null; IAType subType =3D re= cType; boolean nullable =3D false; - for (int i =3D 0; i < = fieldName=2Esize(); i++) { + + List flattenedFieldName =3D A= rrayIndexUtil=2EgetFlattenedKeyFieldNames(unnestList, projectList); + = List unnestFlags =3D ArrayIndexUtil=2EgetUnnestFlags(unnestList,= projectList); + for (int i =3D 0; i < flattenedFieldName=2Esize(); = i++) { if (subType instanceof AUnionType) { n= ullable =3D nullable || ((AUnionType) subType)=2EisUnknownableType(); = subType =3D ((AUnionType) subType)=2EgetActualType(); = } if (subType instanceof ARecordType) { - s= ubType =3D ((ARecordType) subType)=2EgetFieldType(fieldName=2Eget(i)); + = subType =3D ((ARecordType) subType)=2EgetFieldType(flattenedFi= eldName=2Eget(i)); + } else if ((subType instanceof AOrderedLi= stType || subType instanceof AUnorderedListType) - && ar= rayIndicators=2Eget(i - 1) > 0) { - for (int j =3D 0; j < ar= rayIndicators=2Eget(i - 1); j++) { - subType =3D TypeCom= puteUtils=2EextractListItemType(subType); - } + = && unnestFlags=2Eget(i - 1)) { + subType =3D TypeComp= uteUtils=2EextractListItemType(subType); if (subType insta= nceof ARecordType) { - subType =3D ((ARecordType) subTyp= e)=2EgetFieldType(fieldName=2Eget(i)); + subType =3D ((A= RecordType) subType)=2EgetFieldType(flattenedFieldName=2Eget(i)); + = } else { - throw AsterixException=2Ecreate(Err= orCode=2ECOMPILATION_ILLEGAL_STATE, "Unexpected type " + fieldType); + = throw AsterixException=2Ecreate(ErrorCode=2ECOMPILATION_ILLE= GAL_STATE, + "Unexpected type " + subType + ", e= xpected record=2E"); } + } else { - = throw AsterixException=2Ecreate(ErrorCode=2ECOMPILATION_ILLEGAL_STATE= , "Unexpected type " + fieldType); + throw AsterixException= =2Ecreate(ErrorCode=2ECOMPILATION_ILLEGAL_STATE, + "= Unexpected type " + subType + ", expected record, array, or multi-set=2E");= } if (subType =3D=3D null) { @@ -158,18 +169,2= 0 @@ break; } } + if (subTy= pe !=3D null) { - IAType keyType =3D ArrayIndexUtil=2EgetSubFiel= dInArrayType(recType, fieldName, arrayIndicators); + IAType keyT= ype =3D ArrayIndexUtil=2EgetSubFieldType(recType, unnestList, projectList);= Pair pair =3D Index=2EgetNonNullableType(key= Type); - pair=2Esecond =3D pair=2Esecond || ArrayIndexUtil=2EisS= ubFieldNullable(recType, fieldName, arrayIndicators); + pair=2Es= econd =3D pair=2Esecond || ArrayIndexUtil=2EisSubFieldNullable(recType, unn= estList, projectList); keyPairType =3D pair; } + = keyPairType=2Esecond =3D keyPairType=2Esecond || nullable; re= turn keyPairType; } /** - * @deprecated Use new unnestList = and projectList scheme=2E + * @return The concatenation of the unnest l= ist fields and the project field (for use in creating a unique name)=2E = */ public static List getFlattenedKeyFieldNames(List> unnestList, List projectList) { if (unnestList =3D= =3D null) { @@ -188,6 +201,41 @@ } /** + * @return Mapping = to the flattened key field names, determine where the UNNESTs occur=2E + = */ + public static List getUnnestFlags(List> unn= estList, List projectList) { + if (unnestList=2EisEmpty()) {= + // A simple element has no UNNEST flags raised=2E=2E + = List unnestFlags =3D new ArrayList<>(); + for (Str= ing ignored : projectList) { + unnestFlags=2Eadd(false); + = } + return unnestFlags; + + } else { + = List unnestFlagsPrefix =3D new ArrayList<>(); + for = (List unnestField : unnestList) { + for (int i =3D 0= ; i < unnestField=2Esize() - 1; i++) { + unnestFlagsPref= ix=2Eadd(false); + } + unnestFlagsPrefix=2Ead= d(true); + } + + if (projectList =3D=3D null) { + = // Stop here=2E The prefix is the flag vector itself=2E + = return unnestFlagsPrefix; + + } else { + = List unnestFlags =3D new ArrayList<>(unnestFlagsPrefix); + = for (int i =3D 0; i < projectList=2Esize(); i++) { + = unnestFlags=2Eadd(false); + } + return= unnestFlags; + } + } + } + + /** * @deprecat= ed Use new unnestList and projectList scheme=2E */ public static= List getArrayDepthIndicator(List> unnestList, List projectList) { @@ -244,18 +292,14 @@ } /** - * Given= the {@code Index}'s representation of an array path (i=2Ee=2E a concatenat= ion of record paths, with array - * steps specified in depths correspon= ding to an index in the aforementioned record path array), traverse each - = * distinct record path and invoke the appropriate commands for each sce= nario=2E - *

- * Here, we keep track of the record/list type at= each step and give this to each command=2E + * Traverse each distinct = record path and invoke the appropriate commands for each scenario=2E Here, = we keep track + * of the record/list type at each step and give this to= each command=2E */ public static void walkArrayPath(ARecordType= baseRecordType, List flattenedFieldName, - List flattenedDepthIndicators, TypeTrackerCommandExecutor commandExecutor) - = throws AlgebricksException { - ArrayPath arrayPath =3D new= ArrayPath(flattenedFieldName, flattenedDepthIndicators)=2Einvoke(); + = List unnestFlags, TypeTrackerCommandExecutor commandExecuto= r) throws AlgebricksException { + ArrayPath arrayPath =3D new ArrayP= ath(flattenedFieldName, unnestFlags)=2Einvoke(); List= > fieldNamesPerArray =3D arrayPath=2EfieldNamesPerArray; - List depthOfArraySteps =3D arrayPath=2EdepthOfArraySteps; + List unnestFlagsPerArray =3D arrayPath=2EunnestFlagsPerArray; /= / If we are given no base record type, then we do not need to keep track of= the record type=2E We are solely // using this walk for its flag= s=2E @@ -275,7 +319,7 @@ startingStepRecordType)= =2Efirst; } - for (int j =3D 0; j < depthOfArrayS= teps=2Eget(i); j++) { + if (unnestFlagsPerArray=2Eget(i)) { = if (isTrackingType) { workingType =3D Typ= eComputeUtils=2EextractListItemType(workingType); if (= workingType =3D=3D null) { @@ -284,17 +328,14 @@ } = } boolean isFirstArrayStep =3D i =3D=3D 0; -= boolean isFirstUnnestInStep =3D j =3D=3D 0; - = boolean isLastUnnestInIntermediateStep =3D - j =3D= =3D depthOfArraySteps=2Eget(i) - 1 && i < fieldNamesPerArray=2Esize() - 1; = + boolean isLastUnnestInIntermediateStep =3D i < fieldNamesP= erArray=2Esize() - 1; commandExecutor=2EexecuteActionOnEac= hArrayStep(startingStepRecordType, workingType, - fi= eldNamesPerArray=2Eget(i), isFirstArrayStep, isFirstUnnestInStep, - = isLastUnnestInIntermediateStep); + f= ieldNamesPerArray=2Eget(i), isFirstArrayStep, isLastUnnestInIntermediateSte= p); } if (i =3D=3D fieldNamesPerArray=2Esize() = - 1) { - boolean requiresOnlyOneUnnest =3D depthOfArraySteps= =2Estream()=2Ereduce(0, Integer::sum)=2Eequals(1); - boolean= isNonArrayStep =3D depthOfArraySteps=2Eget(i) =3D=3D 0; + b= oolean requiresOnlyOneUnnest =3D fieldNamesPerArray=2Esize() =3D=3D 1; + = boolean isNonArrayStep =3D !unnestFlagsPerArray=2Eget(i); = commandExecutor=2EexecuteActionOnFinalArrayStep(startingStepRe= cordType, fieldNamesPerArray=2Eget(i), isNonArrayS= tep, requiresOnlyOneUnnest); } @@ -302,28 +343,25 @@ } = /** - * Given the {@code Index}'s representation of an array path = (i=2Ee=2E a concatenation of record paths, with array - * steps specifi= ed in depths corresponding to an index in the aforementioned record path ar= ray), traverse each - * distinct record path and invoke the appropriate= commands for each scenario=2E - *

- * Here, we keep track of t= he total number of actions performed and give this to each command=2E + = * Traverse each distinct record path and invoke the appropriate commands f= or each scenario=2E Here, we keep track + * of the total number of acti= ons performed and give this to each command=2E */ - public static = void walkArrayPath(List flattenedFieldName, List flattened= DepthIndicators, + public static void walkArrayPath(List flatten= edFieldName, List unnestFlags, ActionCounterCommandEx= ecutor commandExecutor) throws AlgebricksException { - ArrayPath arr= ayPath =3D new ArrayPath(flattenedFieldName, flattenedDepthIndicators)=2Ein= voke(); + ArrayPath arrayPath =3D new ArrayPath(flattenedFieldName, = unnestFlags)=2Einvoke(); List> fieldNamesPerArray =3D= arrayPath=2EfieldNamesPerArray; - List depthOfArraySteps = =3D arrayPath=2EdepthOfArraySteps; + List unnestFlagsPerArr= ay =3D arrayPath=2EunnestFlagsPerArray; int numberOfActionsPerfo= rmed =3D 0; for (int i =3D 0; i < fieldNamesPerArray=2Esize(); i++= ) { - int unnestLevel =3D depthOfArraySteps=2Eget(i); + = boolean isUnnestFlagRaised =3D unnestFlagsPerArray=2Eget(i); = if (i =3D=3D 0) { commandExecutor=2EexecuteActionOnFirst= ArrayStep(); numberOfActionsPerformed++; - = unnestLevel--; + isUnnestFlagRaised =3D false; = } - for (int j =3D 0; j < unnestLevel; j++) { + if = (isUnnestFlagRaised) { commandExecutor=2EexecuteActionOnIn= termediateArrayStep(numberOfActionsPerformed++); } @@ -343,8= +381,8 @@ public interface TypeTrackerCommandExecutor { vo= id executeActionOnEachArrayStep(ARecordType startingStepRecordType, IAType = workingType, - List fieldName, boolean isFirstArrayS= tep, boolean isFirstUnnestInStep, - boolean isLastUnnestInIn= termediateStep) throws AlgebricksException; + List f= ieldName, boolean isFirstArrayStep, boolean isLastUnnestInIntermediateStep)= + throws AlgebricksException; void executeAction= OnFinalArrayStep(ARecordType startingStepRecordType, List fieldName= , boolean isNonArrayStep, boolean requiresOnlyOneUnnest) t= hrows AlgebricksException; @@ -352,24 +390,24 @@ private static clas= s ArrayPath { private final List flattenedFieldName; - = private final List flattenedDepthIndicators; + private = final List unnestFlags; private List> fieldN= amesPerArray; - private List depthOfArraySteps; + pr= ivate List unnestFlagsPerArray; - public ArrayPath(List flattenedFieldName, List flattenedDepthIndicators) { + = public ArrayPath(List flattenedFieldName, List unnestFl= ags) { this=2EflattenedFieldName =3D flattenedFieldName; - = this=2EflattenedDepthIndicators =3D flattenedDepthIndicators; + = this=2EunnestFlags =3D unnestFlags; } public Ar= rayPath invoke() { fieldNamesPerArray =3D new ArrayList<>(); -= depthOfArraySteps =3D new ArrayList<>(); + unnestFla= gsPerArray =3D new ArrayList<>(); List workingRecordPa= th =3D new ArrayList<>(); - for (int i =3D 0; i < flattenedDepth= Indicators=2Esize(); i++) { + for (int i =3D 0; i < unnestFlags= =2Esize(); i++) { workingRecordPath=2Eadd(flattenedFieldNa= me=2Eget(i)); - if (i =3D=3D flattenedDepthIndicators=2Esi= ze() - 1 || flattenedDepthIndicators=2Eget(i) > 0) { - d= epthOfArraySteps=2Eadd(flattenedDepthIndicators=2Eget(i)); + = if (i =3D=3D unnestFlags=2Esize() - 1 || unnestFlags=2Eget(i)) { + = unnestFlagsPerArray=2Eadd(unnestFlags=2Eget(i)); = fieldNamesPerArray=2Eadd(workingRecordPath); wo= rkingRecordPath =3D new ArrayList<>(); } diff --git a/aste= rixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/KeyF= ieldTypeUtil=2Ejava b/asterixdb/asterix-metadata/src/main/java/org/apache/a= sterix/metadata/utils/KeyFieldTypeUtil=2Ejava index b2026d7=2E=2E6a0c44f 10= 0644 --- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/meta= data/utils/KeyFieldTypeUtil=2Ejava +++ b/asterixdb/asterix-metadata/src/mai= n/java/org/apache/asterix/metadata/utils/KeyFieldTypeUtil=2Ejava @@ -153,9 = +153,7 @@ ARecordType sourceType =3D = (e=2EgetSourceIndicator() =3D=3D Index=2ERECORD_INDICATOR) ? recordType= : metaRecordType; Pair keyPairType =3D A= rrayIndexUtil=2EgetNonNullableOpenFieldType(e=2EgetTypeList()=2Eget(i), - = ArrayIndexUtil=2EgetFlattenedKeyFieldNames(e=2EgetUnn= estList(), e=2EgetProjectList()=2Eget(i)), - sourceT= ype, - ArrayIndexUtil=2EgetArrayDepthIndicator(e=2Eg= etUnnestList(), e=2EgetProjectList()=2Eget(i))); + e= =2EgetUnnestList(), e=2EgetProjectList()=2Eget(i), sourceType); = indexKeyTypes=2Eadd(keyPairType=2Efirst); } } d= iff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/met= adata/utils/SecondaryArrayIndexBTreeOperationsHelper=2Ejava b/asterixdb/ast= erix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryArra= yIndexBTreeOperationsHelper=2Ejava index fdb21a2=2E=2Ec1f8c7b 100644 --- a/= asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/= SecondaryArrayIndexBTreeOperationsHelper=2Ejava +++ b/asterixdb/asterix-met= adata/src/main/java/org/apache/asterix/metadata/utils/SecondaryArrayIndexBT= reeOperationsHelper=2Ejava @@ -57,7 +57,6 @@ import org=2Eapache=2Ehyracks= =2Ealgebricks=2Eruntime=2Ebase=2EIPushRuntimeFactory; import org=2Eapache= =2Ehyracks=2Ealgebricks=2Eruntime=2Ebase=2EIScalarEvaluatorFactory; import= org=2Eapache=2Ehyracks=2Ealgebricks=2Eruntime=2Ebase=2EIUnnestingEvaluator= Factory; -import org=2Eapache=2Ehyracks=2Ealgebricks=2Eruntime=2Eevaluators= =2EColumnAccessEvalFactory; import org=2Eapache=2Ehyracks=2Ealgebricks=2Er= untime=2Eoperators=2Eaggreg=2ESimpleAlgebricksAccumulatingAggregatorFactory= ; import org=2Eapache=2Ehyracks=2Ealgebricks=2Eruntime=2Eoperators=2Ebase= =2ESinkRuntimeFactory; import org=2Eapache=2Ehyracks=2Ealgebricks=2Eruntim= e=2Eoperators=2Emeta=2EAlgebricksMetaOperatorDescriptor; @@ -78,14 +77,13 @= @ public class SecondaryArrayIndexBTreeOperationsHelper extends Secondar= yTreeIndexOperationsHelper { private final int numAtomicSecondaryKeys,= numArraySecondaryKeys, numTotalSecondaryKeys; - private final Index=2EA= rrayIndexDetails arrayIndexDetails; private final EvalFactoryAndRecDes= cStackBuilder evalFactoryAndRecDescStackBuilder =3D new EvalFa= ctoryAndRecDescStackBuilder(); - // TODO (GLENN): Phase these out and = use the UNNEST / PROJECT scheme instead=2E + private final Index=2EArray= IndexDetails arrayIndexDetails; private final List> flatt= enedFieldNames; private final List flattenedKeyTypes; - pri= vate final List> depthIndicators; + private final List> unnestFlags; protected SecondaryArrayIndexBTreeOperation= sHelper(Dataset dataset, Index index, MetadataProvider metadataProvider, = SourceLocation sourceLoc) throws AlgebricksException { @@ -94,19= +92,19 @@ flattenedFieldNames =3D new ArrayList<>(); f= lattenedKeyTypes =3D new ArrayList<>(); - depthIndicators =3D new Ar= rayList<>(); + unnestFlags =3D new ArrayList<>(); for (Inde= x=2EArrayIndexElement e : arrayIndexDetails=2EgetElementList()) { = if (e=2EgetUnnestList()=2EisEmpty()) { flattenedFieldN= ames=2Eadd(e=2EgetProjectList()=2Eget(0)); flattenedKeyTyp= es=2Eadd(e=2EgetTypeList()=2Eget(0)); - depthIndicators - = =2Eadd(ArrayIndexUtil=2EgetArrayDepthIndicator(e=2Eget= UnnestList(), e=2EgetProjectList()=2Eget(0))); + unnestFlags= =2Eadd(ArrayIndexUtil=2EgetUnnestFlags(e=2EgetUnnestList(), e=2EgetProjectL= ist()=2Eget(0))); + } else { for (int i =3D 0= ; i < e=2EgetProjectList()=2Esize(); i++) { List project =3D e=2EgetProjectList()=2Eget(i); flattene= dFieldNames=2Eadd(ArrayIndexUtil=2EgetFlattenedKeyFieldNames(e=2EgetUnnestL= ist(), project)); - depthIndicators=2Eadd(ArrayIndexUtil= =2EgetArrayDepthIndicator(e=2EgetUnnestList(), project)); = flattenedKeyTypes=2Eadd(e=2EgetTypeList()=2Eget(i)); + = unnestFlags=2Eadd(ArrayIndexUtil=2EgetUnnestFlags(e=2EgetUnnestList(), p= roject)); } } } @@ -127,7 +125,7 @@ = numArraySecondaryKeys =3D numTotalSecondaryKeys - numAtomicSeconda= ryKeys; } - private int findPosOfArrayIndex() throws AsterixExcep= tion { + private int findPosOfArrayIndexElement() throws AsterixExceptio= n { for (int i =3D 0; i < arrayIndexDetails=2EgetElementList()=2Es= ize(); i++) { if (!arrayIndexDetails=2EgetElementList()=2Eget(= i)=2EgetUnnestList()=2EisEmpty()) { return i; @@ -163,9 +1= 61,7 @@ ARecordType sourceType =3D (e=2EgetSourceIndicator= () =3D=3D 0) ? itemType : metaType; addSKEvalFactories(isO= verridingKeyFieldTypes ? enforcedItemType : sourceType, flattenedListPos, f= alse); Pair keyTypePair =3D ArrayIndexUti= l=2EgetNonNullableOpenFieldType(e=2EgetTypeList()=2Eget(i), - = ArrayIndexUtil=2EgetFlattenedKeyFieldNames(e=2EgetUnnestList(), e= =2EgetProjectList()=2Eget(i)), - sourceType, - = ArrayIndexUtil=2EgetArrayDepthIndicator(e=2EgetUnnestList= (), e=2EgetProjectList()=2Eget(i))); + e=2EgetUnnest= List(), e=2EgetProjectList()=2Eget(i), sourceType); IAType= keyType =3D keyTypePair=2Efirst; anySecondaryKeyIsNullabl= e =3D anySecondaryKeyIsNullable || keyTypePair=2Esecond; I= SerializerDeserializer keySerde =3D serdeProvider=2EgetSerializerDeserializ= er(keyType); @@ -242,14 +238,15 @@ return; } - = List arrayDepthIndicators =3D depthIndicators=2Eget(fieldPos); = - List fieldNames =3D flattenedFieldNames=2Eget(fieldPos); -= if (arrayDepthIndicators=2Estream()=2EnoneMatch(b -> b > 0)) { + = List flattenedFieldName =3D flattenedFieldNames=2Eget(fieldPos= ); + List workingUnnestFlags =3D unnestFlags=2Eget(fieldPos= ); + if (workingUnnestFlags=2Estream()=2EnoneMatch(b -> b)) { = addAtomicFieldToBuilder(recordType, fieldPos); + } else { = EvalFactoryAndRecDescInvoker commandExecutor =3D = new EvalFactoryAndRecDescInvoker(!evalFactoryAndRecDescStackBuilder= =2EisUnnestEvalPopulated()); - ArrayIndexUtil=2EwalkArrayPath(re= cordType, fieldNames, arrayDepthIndicators, commandExecutor); + = ArrayIndexUtil=2EwalkArrayPath(recordType, flattenedFieldName, workingUnnes= tFlags, commandExecutor); } } @@ -278,13 +275,12 @@ = sourceOp =3D targetOp; } - // TODO (GLE= NN): Refactor to use UNNEST + PROJECT scheme=2E // Perform the= unnest work=2E final Mutable sourceOpRef= =3D new MutableObject<>(sourceOp); final Mutable targetOpRef =3D new MutableObject<>(targetOp); Loadin= gJobBuilder jobBuilder =3D new LoadingJobBuilder(spec, sourceOpRef, targetO= pRef); - int posOfArrayIndex =3D findPosOfArrayIndex(); - = ArrayIndexUtil=2EwalkArrayPath(flattenedFieldNames=2Eget(posOfArrayInd= ex), depthIndicators=2Eget(posOfArrayIndex), + int posOfArrayEle= ment =3D findPosOfArrayIndexElement(); + ArrayIndexUtil=2EwalkAr= rayPath(flattenedFieldNames=2Eget(posOfArrayElement), unnestFlags=2Eget(pos= OfArrayElement), jobBuilder); sourceOp = =3D sourceOpRef=2EgetValue(); @@ -450,8 +446,7 @@ : = inputWidth + numTotalSecondaryKeys + numFilterFields)=2EtoArray(); = for (int i =3D 0; i < numTotalSecondaryKeys; i++) { i= nt sizeOfFieldNamesForI =3D flattenedFieldNames=2Eget(i)=2Esize(); - = if (depthIndicators=2Eget(i)=2Eget(sizeOfFieldNamesForI - 1) !=3D = 0 - && (depthIndicators=2Eget(i)=2Estream()=2EanyMat= ch(b -> b > 0))) { + if (unnestFlags=2Eget(i)=2Eget(sizeOfFi= eldNamesForI - 1)) { projectionList[i] =3D numPrimaryK= eys + 1; } else { projectionList[i] = =3D outColumns[outColumnsCursor++]; @@ -469,9 +464,9 @@ outCol= umns =3D IntStream=2Erange(inputWidth, inputWidth + numArraySecondaryKeys)= =2EtoArray(); for (int i =3D 0; i < numTotalSecondaryKeys; i++= ) { int sizeOfFieldNamesForI =3D flattenedFieldNames=2Eget= (i)=2Esize(); - if (depthIndicators=2Eget(i)=2Estream()=2Eno= neMatch(b -> b > 0)) { + if (unnestFlags=2Eget(i)=2Estream()= =2EnoneMatch(b -> b)) { projectionList[i] =3D numPrima= ryKeys + atomicSKCursor++; - } else if (depthIndicators=2Ege= t(i)=2Eget(sizeOfFieldNamesForI - 1) =3D=3D 0) { + } else if= (!unnestFlags=2Eget(i)=2Eget(sizeOfFieldNamesForI - 1)) { = projectionList[i] =3D outColumns[arraySKCursor++]; } = else { projectionList[i] =3D numPrimaryKeys + numAtomi= cSecondaryKeys + numFilterFields + 1; @@ -525,25 +520,20 @@ @Ove= rride public void executeActionOnEachArrayStep(ARecordType startin= gStepRecordType, IAType workingType, - List fieldNam= e, boolean isFirstArrayStep, boolean isFirstUnnestInStep, - = boolean isLastUnnestInIntermediateStep) throws AlgebricksException { + = List fieldName, boolean isFirstArrayStep, boolean isLast= UnnestInIntermediateStep) + throws AlgebricksException { = if (!this=2EisFirstWalk) { // We have already ad= ded the appropriate UNNESTs=2E return; } = int sourceColumnForNestedArrays =3D numPrimaryKeys + numAtomicSe= condaryKeys + numFilterFields; - if (isFirstUnnestInStep) { - = int sourceColumnForFirstUnnestInAtomicPath =3D - = isFirstArrayStep ? numPrimaryKeys : sourceColumnForNestedArrays; = - IScalarEvaluatorFactory sef =3D metadataProvider=2EgetData= Format()=2EgetFieldAccessEvaluatorFactory( - metadat= aProvider=2EgetFunctionManager(), startingStepRecordType, fieldName, - = sourceColumnForFirstUnnestInAtomicPath, sourceLoc); - = evalFactoryAndRecDescStackBuilder=2EaddUnnest(sef, workingType= ); - } else { - IScalarEvaluatorFactory sef =3D n= ew ColumnAccessEvalFactory(sourceColumnForNestedArrays); - e= valFactoryAndRecDescStackBuilder=2EaddUnnest(sef, workingType); - = } + int sourceColumnForFirstUnnestInAtomicPath =3D + = isFirstArrayStep ? numPrimaryKeys : sourceColumnForNestedArrays; = + IScalarEvaluatorFactory sef =3D metadataProvider=2EgetDataForm= at()=2EgetFieldAccessEvaluatorFactory( + metadataProvide= r=2EgetFunctionManager(), startingStepRecordType, fieldName, + = sourceColumnForFirstUnnestInAtomicPath, sourceLoc); + eva= lFactoryAndRecDescStackBuilder=2EaddUnnest(sef, workingType); } = @Override diff --git a/asterixdb/asterix-metadata/src/main/java/or= g/apache/asterix/metadata/utils/TypeUtil=2Ejava b/asterixdb/asterix-metadat= a/src/main/java/org/apache/asterix/metadata/utils/TypeUtil=2Ejava index dd3= 03fc=2E=2Eb1d610f 100644 --- a/asterixdb/asterix-metadata/src/main/java/org= /apache/asterix/metadata/utils/TypeUtil=2Ejava +++ b/asterixdb/asterix-meta= data/src/main/java/org/apache/asterix/metadata/utils/TypeUtil=2Ejava @@ -61= ,8 +61,8 @@ } private static class EnforcedTypeBuilder { - = private final Deque> typeStack =3D new A= rrayDeque<>(); - private List keyDepthIndicators; + = private final Deque> typeStack =3D new Arra= yDeque<>(); + private List keyUnnestFlags; private= List keyFieldNames; private ARecordType baseRecordType; = private IAType keyFieldType; @@ -72,11 +72,11 @@ private I= AType endOfOpenTypeBuild; private int indexOfOpenPart; - = public void reset(ARecordType baseRecordType, List keyFieldNames, L= ist keyDepthIndicators, + public void reset(ARecordType bas= eRecordType, List keyFieldNames, List keyUnnestFlags, = IAType keyFieldType) { this=2EbaseRecordType =3D = baseRecordType; this=2EkeyFieldNames =3D keyFieldNames; - = this=2EkeyDepthIndicators =3D keyDepthIndicators; + this= =2EkeyUnnestFlags =3D keyUnnestFlags; this=2EkeyFieldType =3D = keyFieldType; } @@ -90,21 +90,19 @@ IAType typeInte= rmediate =3D baseRecordType; List subFieldName =3D new= ArrayList<>(); for (int i =3D 0; i < keyFieldNames=2Esize() -= 1; i++) { - typeStack=2Epush(new Triple<>(typeIntermediate,= keyFieldNames=2Eget(i), - (i =3D=3D 0) ? 0 : keyDep= thIndicators=2Eget(i - 1))); + typeStack=2Epush( + = new Triple<>(typeIntermediate, keyFieldNames=2Eget(i), i !=3D= 0 && keyUnnestFlags=2Eget(i - 1))); bridgeNameFoundFromOp= enTypeBuild =3D typeIntermediate=2EgetTypeName(); - if (i = =3D=3D 0 || keyDepthIndicators=2Eget(i - 1) =3D=3D 0) { + if= (i =3D=3D 0 || !keyUnnestFlags=2Eget(i - 1)) { subFie= ldName=2Eadd(keyFieldNames=2Eget(i)); } else { - = // We have a multi-valued intermediate=2E Traverse the array firs= t, then add our field name=2E - for (int j =3D 0; j < ke= yDepthIndicators=2Eget(i - 1); j++) { - typeIntermed= iate =3D TypeComputeUtils=2EextractListItemType(typeIntermediate); - = if (typeIntermediate =3D=3D null) { - = String fName =3D String=2Ejoin("=2E", subFieldName); - = throw new AsterixException(ErrorCode=2ECOMPILATION_ERROR, - = "Wrong level of array nesting for field: = " + fName); - } + // We have a mu= lti-valued intermediate=2E Perform our UNNEST then add our field name=2E + = typeIntermediate =3D TypeComputeUtils=2EextractListItemT= ype(typeIntermediate); + if (typeIntermediate =3D=3D nul= l) { + String fName =3D String=2Ejoin("=2E", subFiel= dName); + throw new AsterixException(ErrorCode=2ECOM= PILATION_ERROR, + "No list item type found= =2E Wrong type given from field " + fName); } = subFieldName=2Eadd(keyFieldNames=2Eget(i)); }= @@ -133,27 +131,27 @@ } private IAType buildNewForOpen= Type() { - int depthOfOpenType =3D keyDepthIndicators=2EsubList(= indexOfOpenPart + 1, keyDepthIndicators=2Esize())=2Estream() - = =2Efilter(i -> i !=3D 0)=2EfindFirst()=2EorElse(0); - IAT= ype resultant =3D nestArrayType(keyFieldType, depthOfOpenType); + = boolean isTypeWithUnnest =3D keyUnnestFlags=2EsubList(indexOfOpenPart + 1= , keyUnnestFlags=2Esize())=2Estream() + =2Efilter(i -> i= )=2EfindFirst()=2EorElse(false); + IAType resultant =3D nestArra= yType(keyFieldType, isTypeWithUnnest); // Build the type (li= st or record) that holds the type (list or record) above=2E re= sultant =3D nestArrayType( new ARecordType(keyFieldNam= es=2Eget(keyFieldNames=2Esize() - 2), new Stri= ng[] { keyFieldNames=2Eget(keyFieldNames=2Esize() - 1) }, = new IAType[] { AUnionType=2EcreateUnknownableType(resultant) },= true), - keyDepthIndicators=2Eget(indexOfOpenPart)); + = keyUnnestFlags=2Eget(indexOfOpenPart)); /= / Create open part of the nested field=2E for (int i =3D keyFi= eldNames=2Esize() - 3; i > (indexOfOpenPart - 1); i--) { r= esultant =3D nestArrayType( new ARecordType(keyFie= ldNames=2Eget(i), new String[] { keyFieldNames=2Eget(i + 1) }, = new IAType[] { AUnionType=2EcreateUnknownableType(resu= ltant) }, true), - keyDepthIndicators=2Eget(i)); + = keyUnnestFlags=2Eget(i)); } = // Now update the parent to include this optional field, accounting for = intermediate arrays=2E - Triple gapTrip= le =3D this=2EtypeStack=2Epop(); + Triple gapTriple =3D this=2EtypeStack=2Epop(); ARecordType parentR= ecord =3D (ARecordType) unnestArrayType(TypeComputeUti= ls=2EgetActualType(gapTriple=2Efirst), gapTriple=2Ethird); IAT= ype[] parentFieldTypes =3D ArrayUtils=2EaddAll(parentRecord=2EgetFieldTypes= ()=2Eclone(), @@ -168,9 +166,9 @@ private IAType buildNewForFullyC= losedType() throws AsterixException { // The schema is closed = all the way to the field itself=2E IAType typeIntermediate =3D= TypeComputeUtils=2EgetActualType(endOfOpenTypeBuild); - int dep= thOfOpenType =3D (indexOfOpenPart =3D=3D 0) ? 0 : keyDepthIndicators=2Eget(= indexOfOpenPart - 1); - int depthOfKeyType =3D keyDepthIndicator= s=2Eget(indexOfOpenPart); - ARecordType lastNestedRecord =3D (AR= ecordType) unnestArrayType(typeIntermediate, depthOfOpenType); + = boolean isOpenTypeWithUnnest =3D indexOfOpenPart !=3D 0 && keyUnnestFlags= =2Eget(indexOfOpenPart - 1); + boolean isKeyTypeWithUnnest =3D k= eyUnnestFlags=2Eget(indexOfOpenPart); + ARecordType lastNestedRe= cord =3D (ARecordType) unnestArrayType(typeIntermediate, isOpenTypeWithUnne= st); Map recordNameTypesMap =3D createRecordNa= meTypeMap(lastNestedRecord); // If an enforced field already= exists, verify that the type is correct=2E @@ -186,21 +184,21 @@ = } if (enforcedFieldType =3D=3D null) { re= cordNameTypesMap=2Eput(keyFieldNames=2Eget(keyFieldNames=2Esize() - 1), - = AUnionType=2EcreateUnknownableType(nestArrayType(keyF= ieldType, depthOfKeyType))); + AUnionType=2EcreateUn= knownableType(nestArrayType(keyFieldType, isKeyTypeWithUnnest))); = } // Build the nested record, and account for the wrappi= ng array=2E IAType resultant =3D nestArrayType( = new ARecordType(lastNestedRecord=2EgetTypeName(), recordNameTypesMap= =2EkeySet()=2EtoArray(new String[0]), recordNa= meTypesMap=2Evalues()=2EtoArray(new IAType[0]), lastNestedRecord=2EisOpen()= ), - depthOfOpenType); + isOpenTypeWi= thUnnest); return keepUnknown(endOfOpenTypeBuild, resultant); = } private ARecordType buildRestOfRecord(IAType newTypeT= oAdd) { IAType resultant =3D TypeComputeUtils=2EgetActualType(= newTypeToAdd); while (!typeStack=2EisEmpty()) { - = Triple typeFromStack =3D typeStack=2Epop(); + = Triple typeFromStack =3D typeStack= =2Epop(); IAType typeIntermediate =3D unnestArrayType(type= FromStack=2Efirst, typeFromStack=2Ethird); ARecordType rec= ordType =3D (ARecordType) typeIntermediate; IAType[] field= Types =3D recordType=2EgetFieldTypes()=2Eclone(); @@ -228,18 +226,13 @@ = return updatedRecordType; } - private static IA= Type nestArrayType(IAType originalType, int depthOfArrays) { - I= AType resultant =3D originalType; - for (int i =3D 0; i < depthO= fArrays; i++) { - resultant =3D - new= AOrderedListType(resultant, (i =3D=3D depthOfArrays - 1) ? originalType=2E= getTypeName() : null); - } - return resultant; + = private static IAType nestArrayType(IAType originalType, boolean isWithi= nArray) { + return (isWithinArray) ? new AOrderedListType(origin= alType, originalType=2EgetTypeName()) : originalType; } - = private static IAType unnestArrayType(IAType originalType, int depthOfArra= ys) { + private static IAType unnestArrayType(IAType originalType, b= oolean isWithinArray) { IAType resultant =3D originalType; - = for (int i =3D 0; i < depthOfArrays; i++) { + if (isWi= thinArray) { resultant =3D TypeComputeUtils=2EextractListI= temType(resultant); if (resultant !=3D null) { = resultant =3D TypeComputeUtils=2EgetActualType(resultant); @@ -29= 9,7 +292,7 @@ "Indexing an open field is only supp= orted on the record part"); } enforcedTypeBuilder= =2Ereset(enforcedRecordType, keyFieldNames=2Eget(i), - C= ollections=2EnCopies(keyFieldNames=2Eget(i)=2Esize(), 0), keyFieldTypes=2Eg= et(i)); + Collections=2EnCopies(keyFieldNames=2Eget(i)= =2Esize(), false), keyFieldTypes=2Eget(i)); validateRecord(enf= orcedRecordType); enforcedRecordType =3D enforcedTypeBuilder= =2Ebuild(); } @@ -319,7 +312,7 @@ "Indexi= ng an open field is only supported on the record part"); } = enforcedTypeBuilder=2Ereset(enforcedRecordType, keyFieldNames=2Eg= et(i), - Collections=2EnCopies(keyFieldNames=2Eget(i)=2E= size(), 0), keyFieldTypes=2Eget(i)); + Collections=2EnCo= pies(keyFieldNames=2Eget(i)=2Esize(), false), keyFieldTypes=2Eget(i)); = validateRecord(enforcedRecordType); enforcedRecordTyp= e =3D enforcedTypeBuilder=2Ebuild(); } @@ -342,7 +335,7 @@ = List project =3D projectList=2Eget(i); e= nforcedTypeBuilder=2Ereset(enforcedRecordType, Arr= ayIndexUtil=2EgetFlattenedKeyFieldNames(unnestList, project), - = ArrayIndexUtil=2EgetArrayDepthIndicator(unnestList, project), t= ypeList=2Eget(i)); + ArrayIndexUtil=2EgetUnnestFlags= (unnestList, project), typeList=2Eget(i)); validateRecord(= enforcedRecordType); enforcedRecordType =3D enforcedTypeBu= ilder=2Ebuild(); } -- To view, visit https://asterix-gerrit= =2Eics=2Euci=2Eedu/c/asterixdb/+/12063 To unsubscribe, or for help writing = mail filters, visit https://asterix-gerrit=2Eics=2Euci=2Eedu/settings Gerr= it-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Id337d1032796= e1d1c2ce68ea2861b4a81dd19aa5 Gerrit-Change-Number: 12063 Gerrit-PatchSet: 1= Gerrit-Owner: Glenn Galvizo Gerrit-MessageType: newch= ange --/daqNoEYv34= Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Glenn Galvizo has uploaded this change for review=2E

View Change

[NO ISSUE][IDX] Refactoring of array index code=2E
- user model changes: no
- storage format changes: no
- interface ch=
anges: no

Refactor: no longer using depth indicators, rather using U=
NNEST flags with
flattened field names=2E We can no longer have back-to-=
back UNNESTs (i=2Ee=2E arrays
within arrays without an ASSIGN intermedia=
te), as no such array index
can be created=2E=2E

Change-Id: Id337=
d1032796e1d1c2ce68ea2861b4a81dd19aa5
---
M asterixdb/asterix-algebra/=
src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexIns=
ertDeleteRule=2Ejava
M asterixdb/asterix-algebra/src/main/java/org/apach=
e/asterix/optimizer/rules/am/AccessMethodUtils=2Ejava
M asterixdb/asteri=
x-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/ArrayBTreeAcc=
essMethod=2Ejava
M asterixdb/asterix-metadata/src/main/java/org/apache/a=
sterix/metadata/declared/ArrayBTreeResourceFactoryProvider=2Ejava
M aste=
rixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/Arra=
yIndexUtil=2Ejava
M asterixdb/asterix-metadata/src/main/java/org/apache/=
asterix/metadata/utils/KeyFieldTypeUtil=2Ejava
M asterixdb/asterix-metad=
ata/src/main/java/org/apache/asterix/metadata/utils/SecondaryArrayIndexBTre=
eOperationsHelper=2Ejava
M asterixdb/asterix-metadata/src/main/java/org/=
apache/asterix/metadata/utils/TypeUtil=2Ejava
8 files changed, 197 inser=
tions(+), 191 deletions(-)

git pull ssh://asterix-gerrit=2Eics=2E=
uci=2Eedu:29418/asterixdb refs/changes/63/12063/1
diff --git a/aste=
rixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/Intr=
oduceSecondaryIndexInsertDeleteRule=2Ejava b/asterixdb/asterix-algebra/src/=
main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertD=
eleteRule=2Ejava
index c3859be=2E=2E81c6c2a 100644--- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/op=
timizer/rules/IntroduceSecondaryIndexInsertDeleteRule=2Ejava
+++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimize=
r/rules/IntroduceSecondaryIndexInsertDeleteRule=2Ejava
@@ -=
728,10 +728,10 @@
                 // Walk the array path=
=2E
                 List<String> flatFirstFieldName =
=3D ArrayIndexUtil=2EgetFlattenedKeyFieldNames(
           =
              workingElement=2EgetUnnestList(), workingElement=2EgetProject=
List()=2Eget(0));
-    =
            List<Integer> firstArrayIndicator =3D ArrayIndexUtil
-                        =
=2EgetArrayDepthIndicator(workingElement=2EgetUnnestList(), workingElement=
=2EgetProjectList()=2Eget(0));
+                List<Boolean> firstUnnestFlags =3D ArrayIn=
dexUtil=2EgetUnnestFlags(workingElement=2EgetUnnestList(),
+                        workingEleme=
nt=2EgetProjectList()=2Eget(0));
                 ArrayInde=
xUtil=2EwalkArrayPath((isOpenOrNestedField) ? null : recordType, flatFirstF=
ieldName,
-            =
            firstArrayIndicator, branchCreator);
+                        firstUnnestFlags, bran=
chCreator);
 
                 // For all o=
ther elements in the PROJECT list, add an assign=2E
       =
          for (int j =3D 1; j < workingElement=2EgetProjectList()=2Esize=
(); j++) {
@@ -1054,25 +1054,19 @@
 =

         @Override
         public void executeAc=
tionOnEachArrayStep(ARecordType startingStepRecordType, IAType workingType,=

-                List&=
lt;String> fieldName, boolean isFirstArrayStep, boolean isFirstUnnestInS=
tep,
-                b=
oolean isLastUnnestInIntermediateStep) throws AlgebricksException {<=
br>+                List<Str=
ing> fieldName, boolean isFirstArrayStep, boolean isLastUnnestInIntermed=
iateStep)
+          =
      throws AlgebricksException {
             if (!isFirs=
tWalk) {
                 // We have already built the UNNE=
ST path, do not build again=2E
                 return;
             }
 
+            // Get the field we want to UNNEST fr=
om our record=2E
             ILogicalExpression accessToUn=
nestVar;
-            i=
f (isFirstUnnestInStep) {
-                // This is the first UNNEST step=2E Get the field we wa=
nt to UNNEST from our record=2E
-                accessToUnnestVar =3D (startingStepRecordType !=
=3D null)
-            =
            ? getFieldAccessFunction(new MutableObject<>(createLastRe=
cordVarRef()),
-       =
                         startingStepRecordType=2EgetFieldIndex(fieldName=
=2Eget(0)), fieldName)
=
-                        : getFieldAccessFunction(new MutableObject<>=
(createLastRecordVarRef()), -1, fieldName);
-            } else {
-                // This is the second+ UNNEST step=2E =
Refer back to the previously unnested variable=2E
-                accessToUnnestVar =3D new Varia=
bleReferenceExpression(this=2ElastFieldVars=2Eget(0));
-                this=2ElastFieldVars=2Ecle=
ar();
-            }
+            accessToUn=
nestVar =3D (startingStepRecordType !=3D null)
+                    ? getFieldAccessFunction(new=
 MutableObject<>(createLastRecordVarRef()),
+                            startingStepRecor=
dType=2EgetFieldIndex(fieldName=2Eget(0)), fieldName)
+                    : getFieldAccessFunct=
ion(new MutableObject<>(createLastRecordVarRef()), -1, fieldName);
             UnnestingFunctionCallExpression scanCollection =
=3D new UnnestingFunctionCallExpression(
                  =
   BuiltinFunctions=2EgetBuiltinFunctionInfo(BuiltinFunctions=2ESCAN_COLLEC=
TION),
                     Collections=2EsingletonList(new=
 MutableObject<>(accessToUnnestVar)));
diff --git a/a=
sterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/a=
m/AccessMethodUtils=2Ejava b/asterixdb/asterix-algebra/src/main/java/org/ap=
ache/asterix/optimizer/rules/am/AccessMethodUtils=2Ejava
in=
dex c78e89f=2E=2E3f4d3f2 100644
--- a/asterixdb/asterix-alg=
ebra/src/main/java/org/apache/asterix/optimizer/rules/am/AccessMethodUtils=
=2Ejava
+++ b/asterixdb/asterix-algebra/src/main/java/org/a=
pache/asterix/optimizer/rules/am/AccessMethodUtils=2Ejava
@=
@ -3060,7 +3060,6 @@
         MutableInt fieldSource =3D ne=
w MutableInt(0);
         ARecordType workingRecordType =3D=
 subTree=2EgetRecordType();
 
-        // TODO: (GLENN) Refactor this to use Array=
IndexUtil=2E
         // Iterate through our array index st=
ructure=2E We must match the depth and field names for the caller's var=
iable
         // to qualify for an array-index optimizatio=
n=2E
         LogicalVariable varFromParent =3D assignVar;<=
/span>
diff --git a/asterixdb/asterix-algebra/src/main/java/org/ap=
ache/asterix/optimizer/rules/am/ArrayBTreeAccessMethod=2Ejava b/asterixdb/a=
sterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/ArrayBTr=
eeAccessMethod=2Ejava
index 44e4a18=2E=2Ece6b407 100644
--- a/asterixdb/asterix-algebra/src/main/java/org/apache/aster=
ix/optimizer/rules/am/ArrayBTreeAccessMethod=2Ejava
+++ b/a=
sterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/a=
m/ArrayBTreeAccessMethod=2Ejava
@@ -99,7 +99,6 @@ 
     @Override
     protected IATy=
pe getIndexedKeyType(Index=2EIIndexDetails chosenIndexDetails, int keyPos) =
throws CompilationException {
-        // TODO (GLENN): This assumes a flattened key list=2E Refac=
tor / clarify this when removing depth indicators=2E
      =
   Index=2EArrayIndexDetails arrayIndexDetails =3D (Index=2EArrayIndexDetai=
ls) chosenIndexDetails;
         int elementPos =3D 0;
         for (Index=2EArrayIndexElement e : arrayIndexDetails=
=2EgetElementList()) {
diff --git a/asterixdb/asterix-metad=
ata/src/main/java/org/apache/asterix/metadata/declared/ArrayBTreeResourceFa=
ctoryProvider=2Ejava b/asterixdb/asterix-metadata/src/main/java/org/apache/=
asterix/metadata/declared/ArrayBTreeResourceFactoryProvider=2Ejavaindex 5654d16=2E=2Ec0bdc75 100644
--- a/asterixdb/a=
sterix-metadata/src/main/java/org/apache/asterix/metadata/declared/ArrayBTr=
eeResourceFactoryProvider=2Ejava
+++ b/asterixdb/asterix-me=
tadata/src/main/java/org/apache/asterix/metadata/declared/ArrayBTreeResourc=
eFactoryProvider=2Ejava
@@ -18,7 +18,6 @@
 =
 */
 package org=2Eapache=2Easterix=2Emetadata=2Edeclared;<=
/span>
 
-impo=
rt java=2Eutil=2EList;
 import java=2Eutil=2EMap; 
 import org=2Eapache=2Easterix=2Ecommon=2Econfig=
=2EDatasetConfig=2EDatasetType;
@@ -134,10 +133,8 @@=

                 sourceType =3D metaType;
       =
      }
             for (int i =3D 0; i < e=2EgetProjec=
tList()=2Esize(); i++) {
-                =
        ArrayIndexUtil=2EgetFlattenedKeyFieldNames(e=2EgetUnnestList(), pro=
ject), sourceType,
-   =
                     ArrayIndexUtil=2EgetArrayDepthIndicator(e=2EgetUnnestL=
ist(), project));
+  =
                      e=2EgetUnnestList(), e=2EgetProjectList()=2Eget(i), s=
ourceType);
                 IAType keyType =3D keyTypePair=
=2Efirst;
                 secondaryTypeTraits[secondaryTyp=
eTraitPos++] =3D typeTraitProvider=2EgetTypeTrait(keyType);
             }
@@ -175,10 +172,8 @@
      =
           sourceType =3D metaType;
             }             for (int i =3D 0; i < e=2EgetProjectList()=2Esize()=
; i++) {
-             =
   List<String> project =3D e=2EgetProjectList()=2Eget(i);
=
                 Pair<IAType, Boolean> keyTypePair =3D ArrayInd=
exUtil=2EgetNonNullableOpenFieldType(e=2EgetTypeList()=2Eget(i),
=
-                        ArrayInd=
exUtil=2EgetFlattenedKeyFieldNames(e=2EgetUnnestList(), project), sourceTyp=
e,
-                   =
     ArrayIndexUtil=2EgetArrayDepthIndicator(e=2EgetUnnestList(), project))=
;
+                  =
      e=2EgetUnnestList(), e=2EgetProjectList()=2Eget(i), sourceType);
                 IAType keyType =3D keyTypePair=2Efirst;=

                 secondaryCmpFactories[secondaryCmpFactoriesPos++=
] =3D
                         cmpFactoryProvider=2EgetBina=
ryComparatorFactory(keyType, true);
diff --git a/asterixdb/=
asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/ArrayIndex=
Util=2Ejava b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/m=
etadata/utils/ArrayIndexUtil=2Ejava
index f4dfe56=2E=2E0778=
ad9 100644
--- a/asterixdb/asterix-metadata/src/main/java/o=
rg/apache/asterix/metadata/utils/ArrayIndexUtil=2Ejava
+++ =
b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/util=
s/ArrayIndexUtil=2Ejava
@@ -39,14 +39,18 @@
 
 public class ArrayIndexUtil {
     /**<=
/span>
-     * @deprecated Use=
 the project + unnest scheme instead of array indicators=2E
+     * Similar function to Index=
9;s "getSubFieldType", but accounts for array fields as well=2E
      */
-    public static IAType getSubFieldInArrayType(ARecordType recordType, L=
ist<String> subFieldName,
-            List<Integer> arrayDepthIndicators) throws Alge=
bricksException {
-    =
    IAType subType =3D recordType=2EgetFieldType(subFieldName=2Eget(0));
-        for (int i =3D 1=
; i < subFieldName=2Esize(); i++) {
+    public static IAType getSubFieldType(ARecordType rec=
ordType, List<List<String>> unnestList,
+            List<String> projectLis=
t) throws AlgebricksException {
+        List<String> flattenedFieldName =3D ArrayIndexUti=
l=2EgetFlattenedKeyFieldNames(unnestList, projectList);
+        List<Boolean> unnestFlags=
 =3D ArrayIndexUtil=2EgetUnnestFlags(unnestList, projectList);
+        IAType subType =3D recor=
dType=2EgetFieldType(flattenedFieldName=2Eget(0));
+
+        for (int i =3D 1; i < flattenedFieldName=2Esize(); i=
++) {
             if (subType =3D=3D null) {
                 return null;
+
             } else if (subType=2EgetTypeTag(=
)=2Eequals(ATypeTag=2EUNION)) {
                 // Support=
 enforced types here=2E
                 subType =3D ((AUni=
onType) subType)=2EgetActualType();
@@ -56,31 +60,30 @@
                             "Field accessor is not defin=
ed for values of type " + subType=2EgetTypeTag());
   =
              }
             }
-            if (subType=2EgetTypeTag()=2Eequals(AT=
ypeTag=2EOBJECT) && arrayDepthIndicators=2Eget(i - 1) =3D=3D 0) {
-                subType=
 =3D ((ARecordType) subType)=2EgetFieldType(subFieldName=2Eget(i));<=
br>+
+            if (subType=2EgetTypeTag()=2Eequal=
s(ATypeTag=2EOBJECT) && !unnestFlags=2Eget(i - 1)) {
+                subType =3D ((ARec=
ordType) subType)=2EgetFieldType(flattenedFieldName=2Eget(i));
+
             } =
else if ((subType=2EgetTypeTag()=2Eequals(ATypeTag=2EARRAY) || subType=2Ege=
tTypeTag()=2Eequals(ATypeTag=2EMULTISET))
-                    && arrayDepthIndicators=2Eg=
et(i - 1) > 0) {
-  =
              for (int j =3D 0; j < arrayDepthIndicators=2Eget(i - 1); j=
++) {
-                =
    subType =3D TypeComputeUtils=2EextractListItemType(subType);
=
-                }
-                subType =3D (subType=
 !=3D null) ? ((ARecordType) subType)=2EgetFieldType(subFieldName=2Eget(i))=
 : null;
+           =
         && unnestFlags=2Eget(i - 1)) {
+                subType =3D TypeComputeUtils=2E=
extractListItemType(subType);
+                subType =3D (subType !=3D null) ? ((ARecordType) =
subType)=2EgetFieldType(flattenedFieldName=2Eget(i)) : null;
+
             } el=
se {
                 throw new AsterixException(ErrorCode=
=2ECOMPILATION_ERROR,
-=
                        (arrayDepthIndicators=2Eget(i - 1) > 0)-                              =
  ? "Object type given, but array depth indicator is " + "no=
n-zero=2E"
-      =
                          : "Array/multiset type given, but array dept=
h indicator is zero=2E");
+                        unnestFlags=2Eget(i - 1) ? "Object =
type given, but unnest flag is also raised=2E"
+                                : "A=
rray/multiset type given, but unnest flag is lowered=2E");
<=
span>             }
         }
-        if (subType !=3D null && arrayDept=
hIndicators=2Eget(arrayDepthIndicators=2Esize() - 1) > 0) {
+
+        if (subType !=3D null && unnestFlag=
s=2Eget(unnestFlags=2Esize() - 1)) {
             // If the=
 end field is an array, we must extract the list item here as well=2E
-            for (int j =3D =
0; j < arrayDepthIndicators=2Eget(arrayDepthIndicators=2Esize() - 1); j+=
+) {
-                i=
f (subType instanceof AbstractCollectionType) {
-                    subType =3D TypeComputeUtils=
=2EextractListItemType(subType);
-                } else {
-                    throw new AsterixException(ErrorCode=2ECO=
MPILATION_ERROR,
-     =
                       "Array type expected for last term, but given: =
"
-               =
                     + ((subType !=3D null) ? subType=2EgetTypeTag() : &quo=
t;null"));
-      =
          }
+        =
    if (subType instanceof AbstractCollectionType) {
+                subType =3D TypeComputeUt=
ils=2EextractListItemType(subType);
+
+   =
         } else {
+  =
              throw new AsterixException(ErrorCode=2ECOMPILATION_ERROR,
+                       =
 "Array type expected for last term, but given: " + subType=2Eget=
TypeTag());
             }
         }
         return subType;
@@ -88,15 +91,18 @@
 
     /**
      * Given a pat=
h of complex types (i=2Ee=2E lists + records), determine the nullability of=
 the field=2E
-     * @=
deprecated Use the project + unnest scheme instead of array indicators=2E
      */
-    public static boolean isSubFieldNullable(ARecordType recordType, List=
<String> subFieldName,
-            List<Integer> arrayIndicators) throws AlgebricksEx=
ception {
-        IATy=
pe subType =3D recordType=2EgetFieldType(subFieldName=2Eget(0));
=
-        for (int i =3D 1; i <=
 subFieldName=2Esize(); i++) {
+    public static boolean isSubFieldNullable(ARecordType recordT=
ype, List<List<String>> unnestList,
+            List<String> projectList) thr=
ows AlgebricksException {
+        List<String> flattenedFieldName =3D ArrayIndexUtil=2Ege=
tFlattenedKeyFieldNames(unnestList, projectList);
+        List<Boolean> unnestFlags =3D A=
rrayIndexUtil=2EgetUnnestFlags(unnestList, projectList);
+        IAType subType =3D recordType=
=2EgetFieldType(flattenedFieldName=2Eget(0));
+
+        for (int i =3D 1; i < flattenedFieldName=2Esize(); i++) {=

             if (subType =3D=3D null) {
  =
               return true;
             }
+
             if (su=
bType=2EgetTypeTag()=2Eequals(ATypeTag=2EUNION)) {
        =
         if (NonTaggedFormatUtil=2EisOptional(subType)) {
 =
                    return true;
@@ -109,12 +115,12 @@
             }
 
             i=
f (subType instanceof ARecordType) {
-                subType =3D ((ARecordType) subType)=2EgetFie=
ldType(subFieldName=2Eget(i));
-            } else if (subType instanceof AbstractCollectionType &=
amp;& arrayIndicators=2Eget(i - 1) > 0) {
-                for (int j =3D 0; j < arrayIn=
dicators=2Eget(i - 1); j++) {
-                    subType =3D TypeComputeUtils=2EextractListItemT=
ype(subType);
-        =
        }
-            =
    subType =3D (subType !=3D null) ? ((ARecordType) subType)=2EgetFieldTyp=
e(subFieldName=2Eget(i)) : null;
+                subType =3D ((ARecordType) subType)=2EgetField=
Type(flattenedFieldName=2Eget(i));
+
+    =
        } else if (subType instanceof AbstractCollectionType && unn=
estFlags=2Eget(i - 1)) {
+                subType =3D TypeComputeUtils=2EextractListItemType(sub=
Type);
+             =
   subType =3D (subType !=3D null) ? ((ARecordType) subType)=2EgetFieldType=
(flattenedFieldName=2Eget(i)) : null;
+
             } else {
  =
               throw CompilationException=2Ecreate(ErrorCode=2ECOMPILATION_=
ILLEGAL_STATE,
                         "Illegal field=
 type " + subType=2EgetTypeTag() + " when checking field nullabil=
ity");
@@ -125,32 +131,37 @@
 <=
br>     /**
      * Similar function to Index's &=
quot;getNonNullableOpenFieldType", but accounts for array fields as we=
ll=2E
-     * @deprecat=
ed Use the project + unnest scheme instead of array indicators=2E      */
-    pu=
blic static Pair<IAType, Boolean> getNonNullableOpenFieldType(IAType =
fieldType, List<String> fieldName,
-            ARecordType recType, List<Integer> arr=
ayIndicators) throws AlgebricksException {
+    public static Pair<IAType, Boolean> getNon=
NullableOpenFieldType(IAType fieldType, List<List<String>> unne=
stList,
+            =
List<String> projectList, ARecordType recType) throws AlgebricksExcep=
tion {
         Pair<IAType, Boolean> keyPairType =3D=
 null;
         IAType subType =3D recType;
         boolean nullable =3D false;
-        for (int i =3D 0; i < fieldName=2Esize(); i++) {=

+
+        List<String> flattenedFi=
eldName =3D ArrayIndexUtil=2EgetFlattenedKeyFieldNames(unnestList, projectL=
ist);
+        List&l=
t;Boolean> unnestFlags =3D ArrayIndexUtil=2EgetUnnestFlags(unnestList, p=
rojectList);
+       =
 for (int i =3D 0; i < flattenedFieldName=2Esize(); i++) {
             if (subType instanceof AUnionType) {
      =
           nullable =3D nullable || ((AUnionType) subType)=2EisUnknownableT=
ype();
                 subType =3D ((AUnionType) subType)=
=2EgetActualType();
             }
        =
     if (subType instanceof ARecordType) {
-                subType =3D ((ARecordType) subType)=2E=
getFieldType(fieldName=2Eget(i));
+                subType =3D ((ARecordType) subType)=2EgetFiel=
dType(flattenedFieldName=2Eget(i));
+
             } else if ((subType instanceo=
f AOrderedListType || subType instanceof AUnorderedListType)
-                    && array=
Indicators=2Eget(i - 1) > 0) {
-                for (int j =3D 0; j < arrayIndicators=2Eget(=
i - 1); j++) {
-       =
             subType =3D TypeComputeUtils=2EextractListItemType(subType);
-                }
+                    &=
& unnestFlags=2Eget(i - 1)) {
+                subType =3D TypeComputeUtils=2EextractListIte=
mType(subType);
                 if (subType instanceof ARe=
cordType) {
-          =
          subType =3D ((ARecordType) subType)=2EgetFieldType(fieldName=2Ege=
t(i));
+             =
       subType =3D ((ARecordType) subType)=2EgetFieldType(flattenedFieldNam=
e=2Eget(i));
+=

                 } else {
-                    throw AsterixException=2Ecreate(ErrorCode=
=2ECOMPILATION_ILLEGAL_STATE, "Unexpected type " + fieldType);
+                    th=
row AsterixException=2Ecreate(ErrorCode=2ECOMPILATION_ILLEGAL_STATE,=

+                          =
  "Unexpected type " + subType + ", expected record=2E"=
);
                 }
+
             } else {
-                throw AsterixException=
=2Ecreate(ErrorCode=2ECOMPILATION_ILLEGAL_STATE, "Unexpected type &quo=
t; + fieldType);
+   =
             throw AsterixException=2Ecreate(ErrorCode=2ECOMPILATION_ILLEGA=
L_STATE,
+           =
             "Unexpected type " + subType + ", expected reco=
rd, array, or multi-set=2E");
             } 
             if (subType =3D=3D null) {
=
@@ -158,18 +169,20 @@
                 break;<=
br>             }
         }
+
         if (subType !=3D n=
ull) {
-            IAT=
ype keyType =3D ArrayIndexUtil=2EgetSubFieldInArrayType(recType, fieldName,=
 arrayIndicators);
+ =
           IAType keyType =3D ArrayIndexUtil=2EgetSubFieldType(recType, unn=
estList, projectList);
             Pair<IAType, Boolean=
> pair =3D Index=2EgetNonNullableType(keyType);
-            pair=2Esecond =3D pair=2Esecond ||=
 ArrayIndexUtil=2EisSubFieldNullable(recType, fieldName, arrayIndicators);<=
/span>
+            pair=2Es=
econd =3D pair=2Esecond || ArrayIndexUtil=2EisSubFieldNullable(recType, unn=
estList, projectList);
             keyPairType =3D pair;
         }
+
         keyPairType=2Esecond =3D keyPairType=2Eseco=
nd || nullable;
         return keyPairType;
     }
 
     /**
-     * @deprecated Use new unnestList and p=
rojectList scheme=2E
=
+     * @return The concatenation of the unnest list fields and the project=
 field (for use in creating a unique name)=2E
      */
     public static List<String> getFlattenedKeyFieldNames=
(List<List<String>> unnestList, List<String> projectList)=
 {
         if (unnestList =3D=3D null) {
@=
@ -188,6 +201,41 @@
     }
 
     /**
+     * @r=
eturn Mapping to the flattened key field names, determine where the UNNESTs=
 occur=2E
+     */
+    public static List=
<Boolean> getUnnestFlags(List<List<String>> unnestList, L=
ist<String> projectList) {
+        if (unnestList=2EisEmpty()) {
+            // A simple element has no UNNES=
T flags raised=2E=2E
=
+            List<Boolean> unnestFlags =3D new ArrayList<>();
+            for (Stri=
ng ignored : projectList) {
+                unnestFlags=2Eadd(false);
+            }
+            return unnestFlags;
+
+        } else {
+            List<Boolean> unnestFlagsPrefix =3D new Arr=
ayList<>();
+  =
          for (List<String> unnestField : unnestList) {
+                for (int i =3D 0;=
 i < unnestField=2Esize() - 1; i++) {
+                    unnestFlagsPrefix=2Eadd(false);
+                }
+                unnestFl=
agsPrefix=2Eadd(true);
+
+            if (projec=
tList =3D=3D null) {
=
+                // Stop here=2E The prefix is the flag vector itself=2E
+                return=
 unnestFlagsPrefix;
+=

+            } else =
{
+                Li=
st<Boolean> unnestFlags =3D new ArrayList<>(unnestFlagsPrefix);=

+                for=
 (int i =3D 0; i < projectList=2Esize(); i++) {
+                    unnestFlags=2Eadd(false)=
;
+                }<=
/span>
+                retu=
rn unnestFlags;
+    =
        }
+        }<=
/span>
+    }
+
+    /**
      * @deprecated Use new u=
nnestList and projectList scheme=2E
      */
     public static List<Integer> getArrayDepthIndicator(List<Lis=
t<String>> unnestList, List<String> projectList) {@@ -244,18 +292,14 @@
     }
 
     /**
-  =
   * Given the {@code Index}'s representation of an array path (i=2Ee=
=2E a concatenation of record paths, with array
-     * steps specified in depths corresponding to=
 an index in the aforementioned record path array), traverse each-     * distinct record path and=
 invoke the appropriate commands for each scenario=2E
-     * <p>
-     * Here, we keep track of the record/list ty=
pe at each step and give this to each command=2E
+     * Traverse each distinct record path and =
invoke the appropriate commands for each scenario=2E Here, we keep track
+     * of the record/l=
ist type at each step and give this to each command=2E
    =
  */
     public static void walkArrayPath(ARecordType base=
RecordType, List<String> flattenedFieldName,
-            List<Integer> flattenedDepth=
Indicators, TypeTrackerCommandExecutor commandExecutor)
-            throws AlgebricksException {<=
/span>
-        ArrayPath arra=
yPath =3D new ArrayPath(flattenedFieldName, flattenedDepthIndicators)=2Einv=
oke();
+            L=
ist<Boolean> unnestFlags, TypeTrackerCommandExecutor commandExecutor)=
 throws AlgebricksException {
+        ArrayPath arrayPath =3D new ArrayPath(flattenedFieldName,=
 unnestFlags)=2Einvoke();
         List<List<String&g=
t;> fieldNamesPerArray =3D arrayPath=2EfieldNamesPerArray;
-        List<Integer> depthOf=
ArraySteps =3D arrayPath=2EdepthOfArraySteps;
+        List<Boolean> unnestFlagsPerArray =
=3D arrayPath=2EunnestFlagsPerArray;
 
    =
     // If we are given no base record type, then we do not need to keep tr=
ack of the record type=2E We are solely 
         // using =
this walk for its flags=2E
@@ -275,7 +319,7 @@
                         startingStepRecordType)=2Efirst;
             }
 
-            for (int j =3D 0; j < depthOfArraySteps=2Eget(=
i); j++) {
+         =
   if (unnestFlagsPerArray=2Eget(i)) {
                 if =
(isTrackingType) {
                     workingType =3D Typ=
eComputeUtils=2EextractListItemType(workingType);
         =
            if (workingType =3D=3D null) {
@@ -284,17 +328,=
14 @@
                     }
              =
   }
                 boolean isFirstArrayStep =3D i =3D=3D=
 0;
-                bo=
olean isFirstUnnestInStep =3D j =3D=3D 0;
-                boolean isLastUnnestInIntermediateStep =
=3D
-                  =
      j =3D=3D depthOfArraySteps=2Eget(i) - 1 && i < fieldNamesP=
erArray=2Esize() - 1;
+                boolean isLastUnnestInIntermediateStep =3D i < fieldNa=
mesPerArray=2Esize() - 1;
                 commandExecutor=
=2EexecuteActionOnEachArrayStep(startingStepRecordType, workingType,=

-                        fiel=
dNamesPerArray=2Eget(i), isFirstArrayStep, isFirstUnnestInStep,
<=
span style=3D"color: hsl(0, 100%, 40%);">-                        isLastUnn=
estInIntermediateStep);
+                        fieldNamesPerArray=2Eget(i), isFirstArrayStep, =
isLastUnnestInIntermediateStep);
             }
<=
span> 
             if (i =3D=3D fieldNamesPerArray=2Esize(=
) - 1) {
-             =
   boolean requiresOnlyOneUnnest =3D depthOfArraySteps=2Estream()=2Ereduce(=
0, Integer::sum)=2Eequals(1);
-                boolean isNonArrayStep =3D depthOfArraySteps=2Eget(=
i) =3D=3D 0;
+       =
         boolean requiresOnlyOneUnnest =3D fieldNamesPerArray=2Esize() =3D=
=3D 1;
+             =
   boolean isNonArrayStep =3D !unnestFlagsPerArray=2Eget(i);
                 commandExecutor=2EexecuteActionOnFinalArrayStep(starting=
StepRecordType, fieldNamesPerArray=2Eget(i),
              =
           isNonArrayStep, requiresOnlyOneUnnest);
        =
     }
@@ -302,28 +343,25 @@
     }<=
br> 
     /**
-     * Given the {@code Index}'s representation of an arr=
ay path (i=2Ee=2E a concatenation of record paths, with array
-     * steps specified in depths co=
rresponding to an index in the aforementioned record path array), traverse =
each
-     * distinct r=
ecord path and invoke the appropriate commands for each scenario=2E<=
br>-     * <p>
-     * Here, we keep track of the =
total number of actions performed and give this to each command=2E+     * Traverse each distinc=
t record path and invoke the appropriate commands for each scenario=2E Here=
, we keep track
+    =
 * of the total number of actions performed and give this to each command=
=2E
      */
-    public static void walkArrayPath(List<String> flattenedFie=
ldName, List<Integer> flattenedDepthIndicators,
+    public static void walkArrayPath(List=
<String> flattenedFieldName, List<Boolean> unnestFlags,<=
br>             ActionCounterCommandExecutor commandExecutor) throws =
AlgebricksException {
-=
        ArrayPath arrayPath =3D new ArrayPath(flattenedFieldName, flattened=
DepthIndicators)=2Einvoke();
+        ArrayPath arrayPath =3D new ArrayPath(flattenedFieldName, =
unnestFlags)=2Einvoke();
         List<List<String>=
;> fieldNamesPerArray =3D arrayPath=2EfieldNamesPerArray;
-        List<Integer> depthOfA=
rraySteps =3D arrayPath=2EdepthOfArraySteps;
+        List<Boolean> unnestFlagsPerArray =
=3D arrayPath=2EunnestFlagsPerArray;
 
    =
     int numberOfActionsPerformed =3D 0;
         for (int =
i =3D 0; i < fieldNamesPerArray=2Esize(); i++) {
-            int unnestLevel =3D depthOfArra=
ySteps=2Eget(i);
+   =
         boolean isUnnestFlagRaised =3D unnestFlagsPerArray=2Eget(i);
             if (i =3D=3D 0) {
                 =
commandExecutor=2EexecuteActionOnFirstArrayStep();
        =
         numberOfActionsPerformed++;
-                unnestLevel--;
+                isUnnestFlagRaised =3D false;
             }
 
-            for (int j =3D 0; j < unnestLevel; j=
++) {
+            if=
 (isUnnestFlagRaised) {
                 commandExecutor=2E=
executeActionOnIntermediateArrayStep(numberOfActionsPerformed++);             }
 
@@ -343,8 +381,8 @@=

 
     public interface TypeTrackerCommand=
Executor {
         void executeActionOnEachArrayStep(AReco=
rdType startingStepRecordType, IAType workingType,
-                List<String> fieldName, =
boolean isFirstArrayStep, boolean isFirstUnnestInStep,
-                boolean isLastUnnestInInte=
rmediateStep) throws AlgebricksException;
+                List<String> fieldName, boolean=
 isFirstArrayStep, boolean isLastUnnestInIntermediateStep)
+                throws AlgebricksExc=
eption;
 
         void executeActionOnFina=
lArrayStep(ARecordType startingStepRecordType, List<String> fieldName=
,
                 boolean isNonArrayStep, boolean requires=
OnlyOneUnnest) throws AlgebricksException;
@@ -352,24 +390,=
24 @@
 
     private static class ArrayPath=
 {
         private final List<String> flattenedField=
Name;
-        private =
final List<Integer> flattenedDepthIndicators;
+        private final List<Boolean>=
 unnestFlags;
         private List<List<String>&g=
t; fieldNamesPerArray;
=
-        private List<Integer> depthOfArraySteps;
+        private List<Boolean> unn=
estFlagsPerArray;
 
-        public ArrayPath(List<String> flattenedFieldNam=
e, List<Integer> flattenedDepthIndicators) {
+        public ArrayPath(List<String> =
flattenedFieldName, List<Boolean> unnestFlags) {
    =
         this=2EflattenedFieldName =3D flattenedFieldName;
-            this=2EflattenedDepthIndic=
ators =3D flattenedDepthIndicators;
+            this=2EunnestFlags =3D unnestFlags;
<=
span>         }
 
         public ArrayPath=
 invoke() {
             fieldNamesPerArray =3D new ArrayLi=
st<>();
-        =
    depthOfArraySteps =3D new ArrayList<>();
+            unnestFlagsPerArray =3D new Arra=
yList<>();
             List<String> workingRec=
ordPath =3D new ArrayList<>();
-            for (int i =3D 0; i < flattenedDepthIndicator=
s=2Esize(); i++) {
+ =
           for (int i =3D 0; i < unnestFlags=2Esize(); i++) {
=
                 workingRecordPath=2Eadd(flattenedFieldName=2Eget(i))=
;
 
-  =
              if (i =3D=3D flattenedDepthIndicators=2Esize() - 1 || flatten=
edDepthIndicators=2Eget(i) > 0) {
-                    depthOfArraySteps=2Eadd(flattenedDepthIn=
dicators=2Eget(i));
+=
                if (i =3D=3D unnestFlags=2Esize() - 1 || unnestFlags=2Eget(=
i)) {
+              =
      unnestFlagsPerArray=2Eadd(unnestFlags=2Eget(i));
    =
                 fieldNamesPerArray=2Eadd(workingRecordPath);
                     workingRecordPath =3D new ArrayList<>();
                 }
diff --git a/asterixdb/aster=
ix-metadata/src/main/java/org/apache/asterix/metadata/utils/KeyFieldTypeUti=
l=2Ejava b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/meta=
data/utils/KeyFieldTypeUtil=2Ejava
index b2026d7=2E=2E6a0c4=
4f 100644
--- a/asterixdb/asterix-metadata/src/main/java/or=
g/apache/asterix/metadata/utils/KeyFieldTypeUtil=2Ejava
+++=
 b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/uti=
ls/KeyFieldTypeUtil=2Ejava
@@ -153,9 +153,7 @@
                 ARecordType sourceType =3D
           =
              (e=2EgetSourceIndicator() =3D=3D Index=2ERECORD_INDICATOR) ? =
recordType : metaRecordType;
                 Pair<IATyp=
e, Boolean> keyPairType =3D ArrayIndexUtil=2EgetNonNullableOpenFieldType=
(e=2EgetTypeList()=2Eget(i),
-                        ArrayIndexUtil=2EgetFlattenedKeyFieldNames(e=
=2EgetUnnestList(), e=2EgetProjectList()=2Eget(i)),
-                        sourceType,<=
br>-                        Array=
IndexUtil=2EgetArrayDepthIndicator(e=2EgetUnnestList(), e=2EgetProjectList(=
)=2Eget(i)));
+      =
                  e=2EgetUnnestList(), e=2EgetProjectList()=2Eget(i), sourc=
eType);
                 indexKeyTypes=2Eadd(keyPairType=2E=
first);
             }
         }diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/ast=
erix/metadata/utils/SecondaryArrayIndexBTreeOperationsHelper=2Ejava b/aster=
ixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/Secon=
daryArrayIndexBTreeOperationsHelper=2Ejava
index fdb21a2=2E=
=2Ec1f8c7b 100644
--- a/asterixdb/asterix-metadata/src/main=
/java/org/apache/asterix/metadata/utils/SecondaryArrayIndexBTreeOperationsH=
elper=2Ejava
+++ b/asterixdb/asterix-metadata/src/main/java=
/org/apache/asterix/metadata/utils/SecondaryArrayIndexBTreeOperationsHelper=
=2Ejava
@@ -57,7 +57,6 @@
 import org=2Eapa=
che=2Ehyracks=2Ealgebricks=2Eruntime=2Ebase=2EIPushRuntimeFactory; import org=2Eapache=2Ehyracks=2Ealgebricks=2Eruntime=2Ebase=2EISca=
larEvaluatorFactory;
 import org=2Eapache=2Ehyracks=2Ealgeb=
ricks=2Eruntime=2Ebase=2EIUnnestingEvaluatorFactory;
-import org=2Eapache=2Ehyracks=2Ealgebricks=
=2Eruntime=2Eevaluators=2EColumnAccessEvalFactory;
 import =
org=2Eapache=2Ehyracks=2Ealgebricks=2Eruntime=2Eoperators=2Eaggreg=2ESimple=
AlgebricksAccumulatingAggregatorFactory;
 import org=2Eapac=
he=2Ehyracks=2Ealgebricks=2Eruntime=2Eoperators=2Ebase=2ESinkRuntimeFactory=
;
 import org=2Eapache=2Ehyracks=2Ealgebricks=2Eruntime=2Eo=
perators=2Emeta=2EAlgebricksMetaOperatorDescriptor;
@@ -78,=
14 +77,13 @@
 
 public class SecondaryArray=
IndexBTreeOperationsHelper extends SecondaryTreeIndexOperationsHelper {
     private final int numAtomicSecondaryKeys, numArraySeconda=
ryKeys, numTotalSecondaryKeys;
-    private final Index=2EArrayIndexDetails arrayIndexDetails;
     private final EvalFactoryAndRecDescStackBuilder evalFacto=
ryAndRecDescStackBuilder =3D
             new EvalFactoryAn=
dRecDescStackBuilder();
 
-    // TODO (GLENN): Phase these out and use the UNNEST=
 / PROJECT scheme instead=2E
+    private final Index=2EArrayIndexDetails arrayIndexDetails;
     private final List<List<String>> flattenedFie=
ldNames;
     private final List<IAType> flattenedKey=
Types;
-    private fin=
al List<List<Integer>> depthIndicators;
+    private final List<List<Boolean=
>> unnestFlags;
 
     protected Seco=
ndaryArrayIndexBTreeOperationsHelper(Dataset dataset, Index index, Metadata=
Provider metadataProvider,
             SourceLocation sour=
ceLoc) throws AlgebricksException {
@@ -94,19 +92,19 @@
 
         flattenedFieldNames =3D new ArrayLi=
st<>();
         flattenedKeyTypes =3D new ArrayList&=
lt;>();
-        dep=
thIndicators =3D new ArrayList<>();
+        unnestFlags =3D new ArrayList<>();
         for (Index=2EArrayIndexElement e : arrayIndexDetails=
=2EgetElementList()) {
             if (e=2EgetUnnestList()=
=2EisEmpty()) {
                 flattenedFieldNames=2Eadd(=
e=2EgetProjectList()=2Eget(0));
                 flattenedK=
eyTypes=2Eadd(e=2EgetTypeList()=2Eget(0));
-                depthIndicators
-                        =2Eadd(ArrayIndexUt=
il=2EgetArrayDepthIndicator(e=2EgetUnnestList(), e=2EgetProjectList()=2Eget=
(0)));
+             =
   unnestFlags=2Eadd(ArrayIndexUtil=2EgetUnnestFlags(e=2EgetUnnestList(), e=
=2EgetProjectList()=2Eget(0)));
+
             } else {
        =
         for (int i =3D 0; i < e=2EgetProjectList()=2Esize(); i++) {
                     List<String> project =3D e=2EgetPro=
jectList()=2Eget(i);
                     flattenedFieldNam=
es=2Eadd(ArrayIndexUtil=2EgetFlattenedKeyFieldNames(e=2EgetUnnestList(), pr=
oject));
-             =
       depthIndicators=2Eadd(ArrayIndexUtil=2EgetArrayDepthIndicator(e=2Ege=
tUnnestList(), project));
                     flattenedKey=
Types=2Eadd(e=2EgetTypeList()=2Eget(i));
+                    unnestFlags=2Eadd(ArrayIndexUtil=
=2EgetUnnestFlags(e=2EgetUnnestList(), project));
         =
        }
             }
         }<=
br>@@ -127,7 +125,7 @@
         numArraySecondaryKeys=
 =3D numTotalSecondaryKeys - numAtomicSecondaryKeys;
     }=

 
-   =
 private int findPosOfArrayIndex() throws AsterixException {
+    private int findPosOfArrayInde=
xElement() throws AsterixException {
         for (int i =
=3D 0; i < arrayIndexDetails=2EgetElementList()=2Esize(); i++) {<=
br>             if (!arrayIndexDetails=2EgetElementList()=2Eget(i)=2E=
getUnnestList()=2EisEmpty()) {
                 return i;
@@ -163,9 +161,7 @@
                 ARecord=
Type sourceType =3D (e=2EgetSourceIndicator() =3D=3D 0) ? itemType : metaTy=
pe;
                 addSKEvalFactories(isOverridingKeyFiel=
dTypes ? enforcedItemType : sourceType, flattenedListPos, false);                 Pair<IAType, Boolean> keyTypePair =3D ArrayIn=
dexUtil=2EgetNonNullableOpenFieldType(e=2EgetTypeList()=2Eget(i),-                        ArrayIn=
dexUtil=2EgetFlattenedKeyFieldNames(e=2EgetUnnestList(), e=2EgetProjectList=
()=2Eget(i)),
-        =
                sourceType,
-                        ArrayIndexUtil=2EgetArrayDepthIndicator(e=2Eg=
etUnnestList(), e=2EgetProjectList()=2Eget(i)));
+                        e=2EgetUnnestList(), e=
=2EgetProjectList()=2Eget(i), sourceType);
                =
 IAType keyType =3D keyTypePair=2Efirst;
                 a=
nySecondaryKeyIsNullable =3D anySecondaryKeyIsNullable || keyTypePair=2Esec=
ond;
                 ISerializerDeserializer keySerde =3D =
serdeProvider=2EgetSerializerDeserializer(keyType);
@@ -242=
,14 +238,15 @@
             return;
       =
  }
 
-=
        List<Integer> arrayDepthIndicators =3D depthIndicators=2Eget(=
fieldPos);
-        Lis=
t<String> fieldNames =3D flattenedFieldNames=2Eget(fieldPos);<=
br>-        if (arrayDepthIndicat=
ors=2Estream()=2EnoneMatch(b -> b > 0)) {
+        List<String> flattenedFieldName =
=3D flattenedFieldNames=2Eget(fieldPos);
+        List<Boolean> workingUnnestFlags =3D unn=
estFlags=2Eget(fieldPos);
+        if (workingUnnestFlags=2Estream()=2EnoneMatch(b -> b)) {
             addAtomicFieldToBuilder(recordType, fieldPos);<=
/span>
+
   =
      } else {
             EvalFactoryAndRecDescInvoker co=
mmandExecutor =3D
                     new EvalFactoryAndRe=
cDescInvoker(!evalFactoryAndRecDescStackBuilder=2EisUnnestEvalPopulated());=

-            ArrayInde=
xUtil=2EwalkArrayPath(recordType, fieldNames, arrayDepthIndicators, command=
Executor);
+         =
   ArrayIndexUtil=2EwalkArrayPath(recordType, flattenedFieldName, workingUn=
nestFlags, commandExecutor);
         }
   =
  }
 
@@ -278,13 +275,12 @@
                 sourceOp =3D targetOp;
             }
 
-       =
     // TODO (GLENN): Refactor to use UNNEST + PROJECT scheme=2E
=
             // Perform the unnest work=2E
          =
   final Mutable<IOperatorDescriptor> sourceOpRef =3D new MutableObje=
ct<>(sourceOp);
             final Mutable<IOperat=
orDescriptor> targetOpRef =3D new MutableObject<>(targetOp);
             LoadingJobBuilder jobBuilder =3D new LoadingJobBuil=
der(spec, sourceOpRef, targetOpRef);
-            int posOfArrayIndex =3D findPosOfArrayIndex();
-            ArrayIndexU=
til=2EwalkArrayPath(flattenedFieldNames=2Eget(posOfArrayIndex), depthIndica=
tors=2Eget(posOfArrayIndex),
+            int posOfArrayElement =3D findPosOfArrayIndexElement()=
;
+            ArrayI=
ndexUtil=2EwalkArrayPath(flattenedFieldNames=2Eget(posOfArrayElement), unne=
stFlags=2Eget(posOfArrayElement),
                     jobB=
uilder);
             sourceOp =3D sourceOpRef=2EgetValue()=
;
 
@@ -450,8 +446,7 @@
   =
                  : inputWidth + numTotalSecondaryKeys + numFilterFields)=
=2EtoArray();
             for (int i =3D 0; i < numTota=
lSecondaryKeys; i++) {
                 int sizeOfFieldName=
sForI =3D flattenedFieldNames=2Eget(i)=2Esize();
-                if (depthIndicators=2Eget(i)=2Eg=
et(sizeOfFieldNamesForI - 1) !=3D 0
-                        && (depthIndicators=2Eget(i)=
=2Estream()=2EanyMatch(b -> b > 0))) {
+                if (unnestFlags=2Eget(i)=2Eget(siz=
eOfFieldNamesForI - 1)) {
                     projectionLi=
st[i] =3D numPrimaryKeys + 1;
                 } else {
                     projectionList[i] =3D outColumns[outColum=
nsCursor++];
@@ -469,9 +464,9 @@
          =
   outColumns =3D IntStream=2Erange(inputWidth, inputWidth + numArraySecond=
aryKeys)=2EtoArray();
             for (int i =3D 0; i <=
 numTotalSecondaryKeys; i++) {
                 int sizeOfF=
ieldNamesForI =3D flattenedFieldNames=2Eget(i)=2Esize();
-                if (depthIndicators=2Ege=
t(i)=2Estream()=2EnoneMatch(b -> b > 0)) {
+                if (unnestFlags=2Eget(i)=2Estr=
eam()=2EnoneMatch(b -> b)) {
                     projec=
tionList[i] =3D numPrimaryKeys + atomicSKCursor++;
-                } else if (depthIndicators=2Eg=
et(i)=2Eget(sizeOfFieldNamesForI - 1) =3D=3D 0) {
+                } else if (!unnestFlags=2Eget=
(i)=2Eget(sizeOfFieldNamesForI - 1)) {
                    =
 projectionList[i] =3D outColumns[arraySKCursor++];
       =
          } else {
                     projectionList[i] =
=3D numPrimaryKeys + numAtomicSecondaryKeys + numFilterFields + 1;@@ -525,25 +520,20 @@
 
         @O=
verride
         public void executeActionOnEachArrayStep(A=
RecordType startingStepRecordType, IAType workingType,
-                List<String> fieldNa=
me, boolean isFirstArrayStep, boolean isFirstUnnestInStep,
-                boolean isLastUnnestIn=
IntermediateStep) throws AlgebricksException {
+                List<String> fieldName, bo=
olean isFirstArrayStep, boolean isLastUnnestInIntermediateStep)
<=
span style=3D"color: hsl(120, 100%, 40%);">+                throws Algebric=
ksException {
             if (!this=2EisFirstWalk) {
                 // We have already added the appropriate UNNEST=
s=2E
                 return;
             =
}
 
             int sourceColumnForNestedA=
rrays =3D numPrimaryKeys + numAtomicSecondaryKeys + numFilterFields;=

-            if (isFirstUnnes=
tInStep) {
-           =
     int sourceColumnForFirstUnnestInAtomicPath =3D
-                        isFirstArrayStep ? =
numPrimaryKeys : sourceColumnForNestedArrays;
-                IScalarEvaluatorFactory sef =3D met=
adataProvider=2EgetDataFormat()=2EgetFieldAccessEvaluatorFactory(-                        metadat=
aProvider=2EgetFunctionManager(), startingStepRecordType, fieldName,=

-                        sour=
ceColumnForFirstUnnestInAtomicPath, sourceLoc);
-                evalFactoryAndRecDescStackBuilder=
=2EaddUnnest(sef, workingType);
-            } else {
-                IScalarEvaluatorFactory sef =3D new ColumnAccessEv=
alFactory(sourceColumnForNestedArrays);
-                evalFactoryAndRecDescStackBuilder=2EaddUn=
nest(sef, workingType);
-            }
+    =
        int sourceColumnForFirstUnnestInAtomicPath =3D
+                    isFirstArrayStep ? n=
umPrimaryKeys : sourceColumnForNestedArrays;
+            IScalarEvaluatorFactory sef =3D metada=
taProvider=2EgetDataFormat()=2EgetFieldAccessEvaluatorFactory(
+                    metadataProv=
ider=2EgetFunctionManager(), startingStepRecordType, fieldName,
<=
span style=3D"color: hsl(120, 100%, 40%);">+                    sourceColum=
nForFirstUnnestInAtomicPath, sourceLoc);
+            evalFactoryAndRecDescStackBuilder=2EaddUnn=
est(sef, workingType);
         }
 <=
br>         @Override
diff --git a/asterixdb/asterix-=
metadata/src/main/java/org/apache/asterix/metadata/utils/TypeUtil=2Ejava b/=
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/=
TypeUtil=2Ejava
index dd303fc=2E=2Eb1d610f 100644--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/me=
tadata/utils/TypeUtil=2Ejava
+++ b/asterixdb/asterix-metada=
ta/src/main/java/org/apache/asterix/metadata/utils/TypeUtil=2Ejava@@ -61,8 +61,8 @@
     }
      private static class EnforcedTypeBuilder {
-        private final Deque<Triple<=
IAType, String, Integer>> typeStack =3D new ArrayDeque<>();
-        private List<I=
nteger> keyDepthIndicators;
+        private final Deque<Triple<IAType, String, Boolean=
>> typeStack =3D new ArrayDeque<>();
+        private List<Boolean> keyUnnestF=
lags;
         private List<String> keyFieldNames;
         private ARecordType baseRecordType;
=
         private IAType keyFieldType;
@@ -72,11 +72,11 @@
         private IAType endOfOpenTypeBuild;
=
         private int indexOfOpenPart;
 
-        public void reset(ARecordType bas=
eRecordType, List<String> keyFieldNames, List<Integer> keyDepth=
Indicators,
+        =
public void reset(ARecordType baseRecordType, List<String> keyFieldNa=
mes, List<Boolean> keyUnnestFlags,
                 I=
AType keyFieldType) {
             this=2EbaseRecordType =
=3D baseRecordType;
             this=2EkeyFieldNames =3D k=
eyFieldNames;
-        =
    this=2EkeyDepthIndicators =3D keyDepthIndicators;
+            this=2EkeyUnnestFlags =3D key=
UnnestFlags;
             this=2EkeyFieldType =3D keyFieldT=
ype;
         }
 
@@ -90,21=
 +90,19 @@
             IAType typeIntermediate =3D baseRec=
ordType;
             List<String> subFieldName =3D n=
ew ArrayList<>();
             for (int i =3D 0; i &l=
t; keyFieldNames=2Esize() - 1; i++) {
-                typeStack=2Epush(new Triple<>(typeInt=
ermediate, keyFieldNames=2Eget(i),
-                        (i =3D=3D 0) ? 0 : keyDepthIndicators=
=2Eget(i - 1)));
+   =
             typeStack=2Epush(
+                        new Triple<>(typeIntermediate, key=
FieldNames=2Eget(i), i !=3D 0 && keyUnnestFlags=2Eget(i - 1)));
                 bridgeNameFoundFromOpenTypeBuild =3D typeInte=
rmediate=2EgetTypeName();
 
-                if (i =3D=3D 0 || keyDepthIndicators=
=2Eget(i - 1) =3D=3D 0) {
+                if (i =3D=3D 0 || !keyUnnestFlags=2Eget(i - 1)) {
                     subFieldName=2Eadd(keyFieldNames=2Eget(i)=
);
                 } else {
-                    // We have a multi-valued interm=
ediate=2E Traverse the array first, then add our field name=2E
-                    for (int j =3D=
 0; j < keyDepthIndicators=2Eget(i - 1); j++) {
-                        typeIntermediate =3D T=
ypeComputeUtils=2EextractListItemType(typeIntermediate);
-                        if (typeIntermed=
iate =3D=3D null) {
-  =
                          String fName =3D String=2Ejoin("=2E", s=
ubFieldName);
-        =
                    throw new AsterixException(ErrorCode=2ECOMPILATION_ERRO=
R,
-                   =
                 "Wrong level of array nesting for field: " + fNa=
me);
-                 =
       }
+           =
         // We have a multi-valued intermediate=2E Perform our UNNEST then =
add our field name=2E
+                    typeIntermediate =3D TypeComputeUtils=2EextractListIt=
emType(typeIntermediate);
+                    if (typeIntermediate =3D=3D null) {
+                        String fN=
ame =3D String=2Ejoin("=2E", subFieldName);
+                        throw new Asterix=
Exception(ErrorCode=2ECOMPILATION_ERROR,
+                                "No list item typ=
e found=2E Wrong type given from field " + fName);
   =
                  }
                     subFieldName=2Eadd=
(keyFieldNames=2Eget(i));
                 }
@@ -133,27 +131,27 @@
         }
 =

         private IAType buildNewForOpenType() {
-            int depthOfOpenType =3D key=
DepthIndicators=2EsubList(indexOfOpenPart + 1, keyDepthIndicators=2Esize())=
=2Estream()
-          =
          =2Efilter(i -> i !=3D 0)=2EfindFirst()=2EorElse(0);
=
-            IAType resultant =3D=
 nestArrayType(keyFieldType, depthOfOpenType);
+            boolean isTypeWithUnnest =3D keyUnne=
stFlags=2EsubList(indexOfOpenPart + 1, keyUnnestFlags=2Esize())=2Estream()<=
/span>
+                    =
=2Efilter(i -> i)=2EfindFirst()=2EorElse(false);
+            IAType resultant =3D nestArra=
yType(keyFieldType, isTypeWithUnnest);
 
  =
           // Build the type (list or record) that holds the type (list or =
record) above=2E
             resultant =3D nestArrayType(<=
/span>
                     new ARecordType(keyFieldNames=2Eget(ke=
yFieldNames=2Esize() - 2),
                             new=
 String[] { keyFieldNames=2Eget(keyFieldNames=2Esize() - 1) },
                             new IAType[] { AUnionType=2EcreateUnknowna=
bleType(resultant) }, true),
-                    keyDepthIndicators=2Eget(indexOfOpenPart));
+                    keyU=
nnestFlags=2Eget(indexOfOpenPart));
 
     =
        // Create open part of the nested field=2E
        =
     for (int i =3D keyFieldNames=2Esize() - 3; i > (indexOfOpenPart - 1=
); i--) {
                 resultant =3D nestArrayType(
                         new ARecordType(keyFieldNames=2Eget(i=
), new String[] { keyFieldNames=2Eget(i + 1) },
           =
                      new IAType[] { AUnionType=2EcreateUnknownableType(res=
ultant) }, true),
-    =
                    keyDepthIndicators=2Eget(i));
+                        keyUnnestFlags=2Eget(=
i));
             }
 
     =
        // Now update the parent to include this optional field, accounting=
 for intermediate arrays=2E
-            Triple<IAType, String, Integer> gapTriple =3D this=
=2EtypeStack=2Epop();
+            Triple<IAType, String, Boolean> gapTriple =3D this=2Ety=
peStack=2Epop();
             ARecordType parentRecord =3D<=
/span>
                     (ARecordType) unnestArrayType(TypeComp=
uteUtils=2EgetActualType(gapTriple=2Efirst), gapTriple=2Ethird);
=
             IAType[] parentFieldTypes =3D ArrayUtils=2EaddAll(parent=
Record=2EgetFieldTypes()=2Eclone(),
@@ -168,9 +166,9 @@
         private IAType buildNewForFullyClosedType() throws As=
terixException {
             // The schema is closed all t=
he way to the field itself=2E
             IAType typeInter=
mediate =3D TypeComputeUtils=2EgetActualType(endOfOpenTypeBuild);-            int depthOfOpenType=
 =3D (indexOfOpenPart =3D=3D 0) ? 0 : keyDepthIndicators=2Eget(indexOfOpenP=
art - 1);
-            =
int depthOfKeyType =3D keyDepthIndicators=2Eget(indexOfOpenPart);-            ARecordType lastNes=
tedRecord =3D (ARecordType) unnestArrayType(typeIntermediate, depthOfOpenTy=
pe);
+            boo=
lean isOpenTypeWithUnnest =3D indexOfOpenPart !=3D 0 && keyUnnestFl=
ags=2Eget(indexOfOpenPart - 1);
+            boolean isKeyTypeWithUnnest =3D keyUnnestFlags=2Ege=
t(indexOfOpenPart);
+=
            ARecordType lastNestedRecord =3D (ARecordType) unnestArrayType(=
typeIntermediate, isOpenTypeWithUnnest);
             Map&l=
t;String, IAType> recordNameTypesMap =3D createRecordNameTypeMap(lastNes=
tedRecord);
 
             // If an enforce=
d field already exists, verify that the type is correct=2E
=
@@ -186,21 +184,21 @@
             }
      =
       if (enforcedFieldType =3D=3D null) {
               =
  recordNameTypesMap=2Eput(keyFieldNames=2Eget(keyFieldNames=2Esize() - 1),=

-                     =
   AUnionType=2EcreateUnknownableType(nestArrayType(keyFieldType, depthOfKe=
yType)));
+          =
              AUnionType=2EcreateUnknownableType(nestArrayType(keyFieldType=
, isKeyTypeWithUnnest)));
             }
 <=
/span>
             // Build the nested record, and account for th=
e wrapping array=2E
             IAType resultant =3D nestA=
rrayType(
                     new ARecordType(lastNestedRe=
cord=2EgetTypeName(), recordNameTypesMap=2EkeySet()=2EtoArray(new String[0]=
),
                             recordNameTypesMap=2Evalues=
()=2EtoArray(new IAType[0]), lastNestedRecord=2EisOpen()),
-                    depthOfOpenType);<=
/span>
+                    =
isOpenTypeWithUnnest);
             return keepUnknown(endO=
fOpenTypeBuild, resultant);
         }
 
         private ARecordType buildRestOfRecord(IAType newType=
ToAdd) {
             IAType resultant =3D TypeComputeUtils=
=2EgetActualType(newTypeToAdd);
             while (!typeSt=
ack=2EisEmpty()) {
-   =
             Triple<IAType, String, Integer> typeFromStack =3D typeSt=
ack=2Epop();
+       =
         Triple<IAType, String, Boolean> typeFromStack =3D typeStack=
=2Epop();
                 IAType typeIntermediate =3D unne=
stArrayType(typeFromStack=2Efirst, typeFromStack=2Ethird);
=
                 ARecordType recordType =3D (ARecordType) typeIntermediate;=

                 IAType[] fieldTypes =3D recordType=2EgetF=
ieldTypes()=2Eclone();
@@ -228,18 +226,13 @@
             return updatedRecordType;
         }<=
br> 
-        pri=
vate static IAType nestArrayType(IAType originalType, int depthOfArrays) {<=
/span>
-            IAType res=
ultant =3D originalType;
-                resultant =3D
-                        new=
 AOrderedListType(resultant, (i =3D=3D depthOfArrays - 1) ? originalType=2E=
getTypeName() : null);
=
-            }
-       =
     return resultant;
+  =
          return (isWithinArray) ? new AOrderedListType(originalType, origi=
nalType=2EgetTypeName()) : originalType;
         }<=
br> 
-        pri=
vate static IAType unnestArrayType(IAType originalType, int depthOfArrays) =
{
+        private st=
atic IAType unnestArrayType(IAType originalType, boolean isWithinArray) {
             IAType resultant =3D originalType;
-            for (int i =3D 0; i &l=
t; depthOfArrays; i++) {
+            if (isWithinArray) {
                 resu=
ltant =3D TypeComputeUtils=2EextractListItemType(resultant);
                 if (resultant !=3D null) {
             =
        resultant =3D TypeComputeUtils=2EgetActualType(resultant);@@ -299,7 +292,7 @@
                         "=
Indexing an open field is only supported on the record part");<=
br>             }
             enforcedTypeBuilder=2E=
reset(enforcedRecordType, keyFieldNames=2Eget(i),
-                    Collections=2EnCopies(keyFi=
eldNames=2Eget(i)=2Esize(), 0), keyFieldTypes=2Eget(i));
+                    Collections=2EnCop=
ies(keyFieldNames=2Eget(i)=2Esize(), false), keyFieldTypes=2Eget(i));
             validateRecord(enforcedRecordType);
             enforcedRecordType =3D enforcedTypeBuilder=2Ebuild();<=
br>         }
@@ -319,7 +312,7 @@
   =
                      "Indexing an open field is only supported on the=
 record part");
             }
       =
      enforcedTypeBuilder=2Ereset(enforcedRecordType, keyFieldNames=2Eget(i=
),
-                   =
 Collections=2EnCopies(keyFieldNames=2Eget(i)=2Esize(), 0), keyFieldTypes=
=2Eget(i));
+        =
            Collections=2EnCopies(keyFieldNames=2Eget(i)=2Esize(), false), =
keyFieldTypes=2Eget(i));
             validateRecord(enforc=
edRecordType);
             enforcedRecordType =3D enforced=
TypeBuilder=2Ebuild();
         }
@@ -342,7=
 +335,7 @@
                 List<String> project =3D =
projectList=2Eget(i);
                 enforcedTypeBuilder=
=2Ereset(enforcedRecordType,
                         Array=
IndexUtil=2EgetFlattenedKeyFieldNames(unnestList, project),
-                        ArrayIndexUti=
l=2EgetArrayDepthIndicator(unnestList, project), typeList=2Eget(i));=

+                        Ar=
rayIndexUtil=2EgetUnnestFlags(unnestList, project), typeList=2Eget(i));
                 validateRecord(enforcedRecordType);                 enforcedRecordType =3D enforcedTypeBuilder=2Ebuild(=
);
             }

To vi= ew, visit change 12063=2E To unsubscribe, or for help writing mail filter= s, visit sett= ings=2E

= Gerrit-Project: asterixdb
Gerrit-Branch= : master
Gerrit-Change-Id: Id337d103279= 6e1d1c2ce68ea2861b4a81dd19aa5
Gerrit-Ch= ange-Number: 12063
Gerrit-PatchSet: 1 <= /div>
Gerrit-Owner: Glenn Galvizo <ggalvizo= @uci=2Eedu>
Gerrit-MessageType: newc= hange
--/daqNoEYv34=--