asterixdb-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ima...@apache.org
Subject [9/9] incubator-asterixdb git commit: Changed metadata storage format for nullable field types. Moved field name generation to the client out of metadata node code. Changed naming scheme for autogenerated types. Moved GroupName, CompactionPolicy & Compac
Date Thu, 20 Aug 2015 02:27:54 GMT
Changed metadata storage format for nullable field types.
Moved field name generation to the client out of metadata node code.
Changed naming scheme for autogenerated types.
Moved GroupName, CompactionPolicy & CompactionPolicyProperties fields
up from External\InternalDetails to Dataset record type definition

Change-Id: I223aded8aaf80f0688358899c0e8b0d6988fac93
Reviewed-on: https://asterix-gerrit.ics.uci.edu/323
Reviewed-by: Till Westmann <tillw@apache.org>
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>


Project: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/commit/a5895308
Tree: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/tree/a5895308
Diff: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/diff/a5895308

Branch: refs/heads/master
Commit: a5895308d88ab4aab2ba573fd11c083f1d868303
Parents: bb24f4b
Author: Ildar Absalyamov <ildar.absalyamov@gmail.com>
Authored: Mon Aug 17 23:15:41 2015 -0700
Committer: Ian Maxon <imaxon@apache.org>
Committed: Wed Aug 19 19:25:17 2015 -0700

----------------------------------------------------------------------
 .../rules/ByNameToByIndexFieldAccessRule.java   |   2 +-
 .../rules/CheckFilterExpressionTypeRule.java    |   4 +-
 ...eDynamicTypeCastForExternalFunctionRule.java |   6 +-
 .../rules/IntroduceDynamicTypeCastRule.java     |  42 +-
 ...IntroduceSecondaryIndexInsertDeleteRule.java |   9 +-
 .../am/AbstractIntroduceAccessMethodRule.java   |  16 +-
 .../rules/typecast/StaticTypeCastUtil.java      |  21 +-
 .../ics/asterix/translator/TypeTranslator.java  |  21 +-
 .../asterix/aql/translator/AqlTranslator.java   |  85 ++-
 .../file/ExternalIndexingOperations.java        |   8 +-
 .../builtin_type/meta03_builtin_type.1.ddl.aql  |  13 +
 .../meta03_builtin_type.2.update.aql            |   7 +
 .../meta03_builtin_type.3.query.aql             |  11 +
 .../meta03_builtin_type_nullable.1.ddl.aql      |  12 +
 .../meta03_builtin_type_nullable.2.update.aql   |   6 +
 .../meta03_builtin_type_nullable.3.query.aql    |  11 +
 .../ordered_list/meta03_ordered_list.1.ddl.aql  |  13 +
 .../meta03_ordered_list.2.update.aql            |   6 +
 .../meta03_ordered_list.3.query.aql             |  11 +
 .../meta03_ordered_list_nullable.1.ddl.aql      |  13 +
 .../meta03_ordered_list_nullable.2.update.aql   |   6 +
 .../meta03_ordered_list_nullable.3.query.aql    |  11 +
 .../complex_type/record/meta03_record.1.ddl.aql |  13 +
 .../record/meta03_record.2.update.aql           |   6 +
 .../record/meta03_record.3.query.aql            |  11 +
 .../meta03_record_nullable.1.ddl.aql            |  13 +
 .../meta03_record_nullable.2.update.aql         |   6 +
 .../meta03_record_nullable.3.query.aql          |  11 +
 .../meta04_unordered_list.1.ddl.aql             |  13 +
 .../meta04_unordered_list.2.update.aql          |   6 +
 .../meta04_unordered_list.3.query.aql           |  11 +
 .../meta04_unordered_list_nullable.1.ddl.aql    |  13 +
 .../meta04_unordered_list_nullable.2.update.aql |   6 +
 .../meta04_unordered_list_nullable.3.query.aql  |  11 +
 .../queries/basic/meta03/meta03.1.ddl.aql       |  13 -
 .../queries/basic/meta03/meta03.2.update.aql    |   7 -
 .../queries/basic/meta03/meta03.3.query.aql     |  11 -
 .../builtin_type/meta04_builtin_type.1.ddl.aql  |  12 +
 .../meta04_builtin_type.2.update.aql            |   6 +
 .../meta04_builtin_type.3.query.aql             |  11 +
 .../meta04_builtin_type_nullable.1.ddl.aql      |  12 +
 .../meta04_builtin_type_nullable.2.update.aql   |   6 +
 .../meta04_builtin_type_nullable.3.query.aql    |  11 +
 .../ordered_list/meta04_ordered_list.1.ddl.aql  |  13 +
 .../meta04_ordered_list.2.update.aql            |   6 +
 .../meta04_ordered_list.3.query.aql             |  11 +
 .../meta04_ordered_list_nullable.1.ddl.aql      |  13 +
 .../meta04_ordered_list_nullable.2.update.aql   |   6 +
 .../meta04_ordered_list_nullable.3.query.aql    |  11 +
 .../complex_type/record/meta04_record.1.ddl.aql |  13 +
 .../record/meta04_record.2.update.aql           |   6 +
 .../record/meta04_record.3.query.aql            |  11 +
 .../meta04_record_nullable.1.ddl.aql            |  13 +
 .../meta04_record_nullable.2.update.aql         |   6 +
 .../meta04_record_nullable.3.query.aql          |  11 +
 .../meta04_unordered_list.1.ddl.aql             |  13 +
 .../meta04_unordered_list.2.update.aql          |   6 +
 .../meta04_unordered_list.3.query.aql           |  11 +
 .../meta04_unordered_list_nullable.1.ddl.aql    |  13 +
 .../meta04_unordered_list_nullable.2.update.aql |   6 +
 .../meta04_unordered_list_nullable.3.query.aql  |  11 +
 .../queries/basic/meta04/meta04.1.ddl.aql       |  12 -
 .../queries/basic/meta04/meta04.2.update.aql    |   6 -
 .../queries/basic/meta04/meta04.3.query.aql     |  11 -
 .../results/basic/issue_251_dataset_hint_2.adm  |   2 +-
 .../issue_251_dataset_hint_2.1.adm              |   2 +-
 .../results/basic/issue_251_dataset_hint_3.adm  |   2 +-
 .../issue_251_dataset_hint_3.1.adm              |   2 +-
 .../results/basic/issue_251_dataset_hint_4.adm  |   2 +-
 .../issue_251_dataset_hint_4.1.adm              |   2 +-
 .../metadata/results/basic/meta02/meta02.1.adm  |   2 +-
 .../builtin_type/meta03_builtin_type.1.adm      |   2 +
 .../meta03_builtin_type_nullable.1.adm          |   2 +
 .../ordered_list/meta03_ordered_list.1.adm      |   3 +
 .../meta03_ordered_list_nullable.1.adm          |   3 +
 .../complex_type/record/meta03_record.1.adm     |   3 +
 .../meta03_record_nullable.1.adm                |   3 +
 .../unordered_list/meta03_unordered_list.1.adm  |   3 +
 .../meta03_unordered_list_nullable.1.adm        |   3 +
 .../metadata/results/basic/meta03/meta03.1.adm  |   2 -
 .../builtin_type/meta04_builtin_type.1.adm      |   2 +
 .../meta04_builtin_type_nullable.1.adm          |   2 +
 .../ordered_list/meta04_ordered_list.1.adm      |   3 +
 .../meta04_ordered_list_nullable.1.adm          |   3 +
 .../complex_type/record/meta04_record.1.adm     |   3 +
 .../meta04_record_nullable.1.adm                |   3 +
 .../unordered_list/meta04_unordered_list.1.adm  |   3 +
 .../meta04_unordered_list_nullable.1.adm        |   3 +
 .../metadata/results/basic/meta04/meta04.1.adm  |   2 -
 .../metadata/results/basic/meta09/meta09.1.adm  |   2 +-
 .../metadata/results/basic/meta16/meta16.1.adm  |  26 +-
 .../metadata/results/basic/meta17/meta17.1.adm  | 133 +++--
 .../resources/metadata/results/basic/meta19.adm |   2 +
 .../metadata/results/basic/meta19/meta19.1.adm  |  32 +-
 .../metadata/results/basic/meta22/meta22.1.adm  |   2 +-
 .../metadata/results/basic/meta23/meta23.1.adm  |   2 +-
 .../metadata_dataset/metadata_dataset.1.adm     |  26 +-
 .../metadata_datatype/metadata_datatype.1.adm   | 133 +++--
 .../basic/metadata_index/metadata_index.1.adm   |  32 +-
 .../verify_failure_previous_success.1.adm       |  23 +-
 ...verify_failure_subsequent_no_execution.1.adm |   2 +-
 .../src/test/resources/metadata/testsuite.xml   |  78 ++-
 .../text-format/text-format.3.query.aql         |   1 +
 .../cross-dataverse/cross-dv02/cross-dv02.1.adm |   8 +-
 .../cross-dataverse/cross-dv04/cross-dv04.1.adm |   8 +-
 .../cross-dataverse/cross-dv19/cross-dv19.1.adm |  14 +-
 .../create-drop-cltype/create-drop-cltype.1.adm |   2 +-
 .../create-drop-opntype.1.adm                   |   2 +-
 .../drop-empty-secondary-indexes.1.adm          |   6 +-
 .../user-defined-functions/udf23/udf23.1.adm    |  12 +-
 .../ics/asterix/aql/expression/DatasetDecl.java |  23 +-
 .../aql/expression/ExternalDetailsDecl.java     |  29 --
 .../asterix/aql/expression/FeedDetailsDecl.java |   5 +-
 .../aql/expression/IDatasetDetailsDecl.java     |   7 -
 .../aql/expression/InternalDetailsDecl.java     |  28 +-
 asterix-aql/src/main/javacc/AQL.jj              |  16 +-
 asterix-doc/src/site/markdown/aql/manual.md     |   4 +-
 .../factory/HDFSIndexingAdapterFactory.java     |   7 +-
 .../factory/StreamBasedAdapterFactory.java      |   2 -
 .../indexing/dataflow/HiveObjectParser.java     | 285 +++++-----
 .../external/library/java/JObjectUtil.java      |   5 +-
 .../ics/asterix/metadata/IDatasetDetails.java   |   7 -
 .../uci/ics/asterix/metadata/MetadataNode.java  | 109 ++--
 .../metadata/MetadataTransactionContext.java    |   2 +-
 .../metadata/bootstrap/MetadataBootstrap.java   |  14 +-
 .../bootstrap/MetadataPrimaryIndexes.java       |  62 ++-
 .../metadata/bootstrap/MetadataRecordTypes.java | 161 +++---
 .../bootstrap/MetadataSecondaryIndexes.java     |  15 +-
 .../AqlCompiledMetadataDeclarations.java        |   6 +-
 .../metadata/declared/AqlMetadataProvider.java  | 514 +++++++++----------
 .../metadata/declared/DatasetDataSource.java    |   2 +-
 .../metadata/declared/LoadableDataSource.java   |   2 +-
 .../entities/AsterixBuiltinTypeMap.java         |   8 +-
 .../ics/asterix/metadata/entities/Dataset.java  |  21 +-
 .../entities/ExternalDatasetDetails.java        |  90 +---
 .../ics/asterix/metadata/entities/Index.java    |  12 +-
 .../entities/InternalDatasetDetails.java        |  56 +-
 .../DatasetTupleTranslator.java                 | 126 ++---
 .../DatatypeTupleTranslator.java                | 189 +++----
 .../IndexTupleTranslator.java                   |  25 +-
 .../asterix/metadata/utils/DatasetUtils.java    |  51 +-
 .../uci/ics/asterix/builders/RecordBuilder.java |   5 +-
 .../data/common/AqlExpressionTypeComputer.java  |   8 +-
 .../AqlPartialAggregationTypeComputer.java      |   9 +-
 .../printers/ANullableFieldPrinterFactory.java  |   2 +-
 .../csv/ANullableFieldPrinterFactory.java       |   4 +-
 .../json/ANullableFieldPrinterFactory.java      |   4 +-
 .../serde/ARecordSerializerDeserializer.java    |   9 +-
 .../nontagged/AqlCSVPrinterFactoryProvider.java |   3 +-
 .../AqlJSONPrinterFactoryProvider.java          |   3 +-
 .../nontagged/AqlPrinterFactoryProvider.java    |   3 +-
 .../pointables/ARecordVisitablePointable.java   |  11 +-
 .../om/pointables/cast/ARecordCaster.java       |  11 +-
 .../pointables/nonvisitor/ARecordPointable.java |   4 +-
 .../GetOverlappingInvervalTypeComputer.java     |   8 +-
 .../impl/InjectFailureTypeComputer.java         |   6 +-
 .../NonTaggedCollectionMemberResultType.java    |   4 +-
 .../NonTaggedFieldAccessByNameResultType.java   |   2 +-
 .../impl/NonTaggedGetItemResultType.java        |  12 +-
 .../impl/NonTaggedLocalAvgTypeComputer.java     |   8 +-
 .../impl/NonTaggedMinMaxAggTypeComputer.java    |  37 +-
 ...onTaggedNumericAddSubMulDivTypeComputer.java |  94 ++--
 .../impl/NonTaggedNumericAggTypeComputer.java   |  26 +-
 ...ggedNumericRoundHalfToEven2TypeComputer.java |  25 +-
 ...nTaggedNumericUnaryFunctionTypeComputer.java |  26 +-
 .../impl/NonTaggedUnaryMinusTypeComputer.java   |  17 +-
 .../impl/OptionalABinaryTypeComputer.java       |  13 +-
 .../impl/OptionalABooleanTypeComputer.java      |   8 +-
 .../impl/OptionalACircleTypeComputer.java       |   8 +-
 .../impl/OptionalADateTimeTypeComputer.java     |   8 +-
 .../impl/OptionalADateTypeComputer.java         |   8 +-
 .../OptionalADayTimeDurationTypeComputer.java   |   8 +-
 .../impl/OptionalADoubleTypeComputer.java       |   8 +-
 .../impl/OptionalADurationTypeComputer.java     |   8 +-
 .../impl/OptionalAFloatTypeComputer.java        |   8 +-
 .../impl/OptionalAInt16TypeComputer.java        |   8 +-
 .../impl/OptionalAInt32TypeComputer.java        |   8 +-
 .../impl/OptionalAInt64TypeComputer.java        |   8 +-
 .../impl/OptionalAInt8TypeComputer.java         |   8 +-
 .../impl/OptionalAIntervalTypeComputer.java     |   8 +-
 .../impl/OptionalALineTypeComputer.java         |   8 +-
 .../impl/OptionalAPoint3DTypeComputer.java      |   8 +-
 .../impl/OptionalAPointTypeComputer.java        |   8 +-
 .../impl/OptionalAPolygonTypeComputer.java      |   8 +-
 .../impl/OptionalARectangleTypeComputer.java    |   8 +-
 .../impl/OptionalAStringTypeComputer.java       |   8 +-
 .../impl/OptionalATimeTypeComputer.java         |   8 +-
 .../OptionalAYearMonthDurationTypeComputer.java |   8 +-
 .../impl/OrderedListConstructorResultType.java  |   5 +-
 .../impl/RecordMergeTypeComputer.java           |   2 +-
 .../impl/Substring2TypeComputer.java            |  10 +-
 .../impl/SubstringTypeComputer.java             |  15 +-
 .../UnaryBinaryInt64OrNullTypeComputer.java     |  16 +-
 .../UnaryBooleanOrNullFunctionTypeComputer.java |   8 +-
 .../UnaryStringInt64OrNullTypeComputer.java     |   4 +-
 .../UnorderedListConstructorResultType.java     |   5 +-
 .../uci/ics/asterix/om/types/ARecordType.java   |  15 +-
 .../uci/ics/asterix/om/types/AUnionType.java    |  29 +-
 .../om/types/AbstractCollectionType.java        |   8 +
 .../asterix/om/types/AbstractComplexType.java   |   2 +
 .../asterix/om/util/NonTaggedFormatUtil.java    |  36 +-
 .../records/FieldAccessByIndexEvalFactory.java  |   5 +-
 .../functions/records/FieldAccessUtil.java      |   7 +-
 .../runtime/formats/NonTaggedDataFormat.java    |   5 +-
 .../runtime/operators/file/ADMDataParser.java   |   8 +-
 .../operators/file/DelimitedDataParser.java     |   6 +-
 .../ics/asterix/tools/datagen/AdmDataGen.java   |  10 +-
 207 files changed, 2006 insertions(+), 1955 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/ByNameToByIndexFieldAccessRule.java
----------------------------------------------------------------------
diff --git a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/ByNameToByIndexFieldAccessRule.java b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/ByNameToByIndexFieldAccessRule.java
index 2cd26d0..0dee2be 100644
--- a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/ByNameToByIndexFieldAccessRule.java
+++ b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/ByNameToByIndexFieldAccessRule.java
@@ -114,7 +114,7 @@ public class ByNameToByIndexFieldAccessRule implements IAlgebraicRewriteRule {
                     case UNION: {
                         AUnionType unionT = (AUnionType) t;
                         if (unionT.isNullableType()) {
-                            IAType t2 = unionT.getUnionList().get(1);
+                            IAType t2 = unionT.getNullableType();
                             if (t2.getTypeTag() == ATypeTag.RECORD) {
                                 ARecordType recType = (ARecordType) t2;
                                 ILogicalExpression fai = createFieldAccessByIndex(recType, fce);

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/CheckFilterExpressionTypeRule.java
----------------------------------------------------------------------
diff --git a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/CheckFilterExpressionTypeRule.java b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/CheckFilterExpressionTypeRule.java
index eaef39a..7e9ebcc 100644
--- a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/CheckFilterExpressionTypeRule.java
+++ b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/CheckFilterExpressionTypeRule.java
@@ -71,8 +71,8 @@ public class CheckFilterExpressionTypeRule implements IAlgebraicRewriteRule {
      * @return true if it is; false otherwise.
      */
     private boolean isPossibleBoolean(IAType type) {
-        while (type.getTypeTag() == ATypeTag.UNION && NonTaggedFormatUtil.isOptionalField((AUnionType) type)) {
-            type = ((AUnionType) type).getUnionList().get(AUnionType.OPTIONAL_TYPE_INDEX_IN_UNION_LIST);
+        while (NonTaggedFormatUtil.isOptional(type)) {
+            type = ((AUnionType) type).getNullableType();
             if (type.getTypeTag() == ATypeTag.BOOLEAN || type.getTypeTag() == ATypeTag.ANY) {
                 return true;
             }

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceDynamicTypeCastForExternalFunctionRule.java
----------------------------------------------------------------------
diff --git a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceDynamicTypeCastForExternalFunctionRule.java b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceDynamicTypeCastForExternalFunctionRule.java
index 76bfcbb..545a336 100644
--- a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceDynamicTypeCastForExternalFunctionRule.java
+++ b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceDynamicTypeCastForExternalFunctionRule.java
@@ -25,6 +25,7 @@ import edu.uci.ics.asterix.om.functions.AsterixBuiltinFunctions;
 import edu.uci.ics.asterix.om.types.ARecordType;
 import edu.uci.ics.asterix.om.types.AUnionType;
 import edu.uci.ics.asterix.om.types.IAType;
+import edu.uci.ics.asterix.om.util.NonTaggedFormatUtil;
 import edu.uci.ics.hyracks.algebricks.common.exceptions.AlgebricksException;
 import edu.uci.ics.hyracks.algebricks.core.algebra.base.ILogicalExpression;
 import edu.uci.ics.hyracks.algebricks.core.algebra.base.ILogicalOperator;
@@ -117,10 +118,9 @@ public class IntroduceDynamicTypeCastForExternalFunctionRule implements IAlgebra
 
         /** the input record type can be an union type -- for the case when it comes from a subplan or left-outer join */
         boolean checkNull = false;
-        while (IntroduceDynamicTypeCastRule.isOptional(inputRecordType)) {
+        while (NonTaggedFormatUtil.isOptional(inputRecordType)) {
             /** while-loop for the case there is a nested multi-level union */
-            inputRecordType = ((AUnionType) inputRecordType).getUnionList().get(
-                    AUnionType.OPTIONAL_TYPE_INDEX_IN_UNION_LIST);
+            inputRecordType = ((AUnionType) inputRecordType).getNullableType();
             checkNull = true;
         }
 

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceDynamicTypeCastRule.java
----------------------------------------------------------------------
diff --git a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceDynamicTypeCastRule.java b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceDynamicTypeCastRule.java
index 6bcd5f0..64c34ed 100644
--- a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceDynamicTypeCastRule.java
+++ b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceDynamicTypeCastRule.java
@@ -112,8 +112,7 @@ public class IntroduceDynamicTypeCastRule implements IAlgebraicRewriteRule {
                     List<LogicalVariable> payloadVars = new ArrayList<LogicalVariable>();
                     expr.getUsedVariables(payloadVars);
                     recordVar = payloadVars.get(0);
-                }
-                else {
+                } else {
                     return false;
                 }
 
@@ -133,13 +132,15 @@ public class IntroduceDynamicTypeCastRule implements IAlgebraicRewriteRule {
                 // of the singular input operator of the DISTRIBUTE_RESULT
                 if (op.getInputs().size() > 1) {
                     // Hopefully not possible?
-                    throw new AlgebricksException("output-record-type defined for expression with multiple input operators");
+                    throw new AlgebricksException(
+                            "output-record-type defined for expression with multiple input operators");
                 }
                 AbstractLogicalOperator input = (AbstractLogicalOperator) op.getInputs().get(0).getValue();
                 List<LogicalVariable> liveVars = new ArrayList<LogicalVariable>();
                 VariableUtilities.getLiveVariables(input, liveVars);
                 if (liveVars.size() > 1) {
-                    throw new AlgebricksException("Expression with multiple fields cannot be cast to output-record-type!");
+                    throw new AlgebricksException(
+                            "Expression with multiple fields cannot be cast to output-record-type!");
                 }
                 recordVar = liveVars.get(0);
 
@@ -156,10 +157,9 @@ public class IntroduceDynamicTypeCastRule implements IAlgebraicRewriteRule {
 
         /** the input record type can be an union type -- for the case when it comes from a subplan or left-outer join */
         boolean checkNull = false;
-        while (isOptional(inputRecordType)) {
+        while (NonTaggedFormatUtil.isOptional(inputRecordType)) {
             /** while-loop for the case there is a nested multi-level union */
-            inputRecordType = ((AUnionType) inputRecordType).getUnionList().get(
-                    AUnionType.OPTIONAL_TYPE_INDEX_IN_UNION_LIST);
+            inputRecordType = ((AUnionType) inputRecordType).getNullableType();
             checkNull = true;
         }
 
@@ -167,12 +167,10 @@ public class IntroduceDynamicTypeCastRule implements IAlgebraicRewriteRule {
         boolean cast = !compatible(requiredRecordType, inputRecordType);
 
         if (checkNull) {
-            recordVar = addWrapperFunction(requiredRecordType, recordVar, op, context,
-                    AsterixBuiltinFunctions.NOT_NULL);
+            recordVar = addWrapperFunction(requiredRecordType, recordVar, op, context, AsterixBuiltinFunctions.NOT_NULL);
         }
         if (cast) {
-            addWrapperFunction(requiredRecordType, recordVar, op, context,
-                    AsterixBuiltinFunctions.CAST_RECORD);
+            addWrapperFunction(requiredRecordType, recordVar, op, context, AsterixBuiltinFunctions.CAST_RECORD);
         }
         return cast || checkNull;
     }
@@ -270,18 +268,16 @@ public class IntroduceDynamicTypeCastRule implements IAlgebraicRewriteRule {
                 return false;
             }
             IAType reqTypeInside = reqTypes[i];
-            if (isOptional(reqTypes[i])) {
-                reqTypeInside = ((AUnionType) reqTypes[i]).getUnionList().get(
-                        AUnionType.OPTIONAL_TYPE_INDEX_IN_UNION_LIST);
+            if (NonTaggedFormatUtil.isOptional(reqTypes[i])) {
+                reqTypeInside = ((AUnionType) reqTypes[i]).getNullableType();
             }
             IAType inputTypeInside = inputTypes[i];
-            if (isOptional(inputTypes[i])) {
-                if (!isOptional(reqTypes[i])) {
+            if (NonTaggedFormatUtil.isOptional(inputTypes[i])) {
+                if (!NonTaggedFormatUtil.isOptional(reqTypes[i])) {
                     /** if the required type is not optional, the two types are incompatible */
                     return false;
                 }
-                inputTypeInside = ((AUnionType) inputTypes[i]).getUnionList().get(
-                        AUnionType.OPTIONAL_TYPE_INDEX_IN_UNION_LIST);
+                inputTypeInside = ((AUnionType) inputTypes[i]).getNullableType();
             }
             if (inputTypeInside.getTypeTag() != ATypeTag.NULL && !reqTypeInside.equals(inputTypeInside)) {
                 return false;
@@ -289,14 +285,4 @@ public class IntroduceDynamicTypeCastRule implements IAlgebraicRewriteRule {
         }
         return true;
     }
-
-    /**
-     * Decide whether a type is an optional type
-     *
-     * @param type
-     * @return true if it is optional; false otherwise
-     */
-    public static boolean isOptional(IAType type) {
-        return type.getTypeTag() == ATypeTag.UNION && NonTaggedFormatUtil.isOptionalField((AUnionType) type);
-    }
 }

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java
----------------------------------------------------------------------
diff --git a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java
index 84f7230..42b60d5 100644
--- a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java
+++ b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java
@@ -570,7 +570,7 @@ public class IntroduceSecondaryIndexInsertDeleteRule implements IAlgebraicRewrit
         // condition.
         for (LogicalVariable secondaryKeyVar : secondaryKeyVars) {
             IAType secondaryKeyType = (IAType) typeEnv.getVarType(secondaryKeyVar);
-            if (!isNullableType(secondaryKeyType) && !forceFilter) {
+            if (!NonTaggedFormatUtil.isOptional(secondaryKeyType) && !forceFilter) {
                 continue;
             }
             ScalarFunctionCallExpression isNullFuncExpr = new ScalarFunctionCallExpression(
@@ -595,11 +595,4 @@ public class IntroduceSecondaryIndexInsertDeleteRule implements IAlgebraicRewrit
         }
         return filterExpression;
     }
-
-    private boolean isNullableType(IAType type) {
-        if (type.getTypeTag() == ATypeTag.UNION) {
-            return ((AUnionType) type).isNullableType();
-        }
-        return false;
-    }
 }

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/am/AbstractIntroduceAccessMethodRule.java
----------------------------------------------------------------------
diff --git a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/am/AbstractIntroduceAccessMethodRule.java b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/am/AbstractIntroduceAccessMethodRule.java
index 0e4270a..b6d07a3 100644
--- a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/am/AbstractIntroduceAccessMethodRule.java
+++ b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/am/AbstractIntroduceAccessMethodRule.java
@@ -188,6 +188,16 @@ public abstract class AbstractIntroduceAccessMethodRule implements IAlgebraicRew
             boolean foundKeyField = false;
             matchedExpressions.clear();
             numMatchedKeys = 0;
+
+            // Remove the candidate if the dataset is a metadata dataset and the index is secondary
+            // TODO: fix the way secondary metadata indexes are implemented and remove this check
+            if (accessMethod.matchPrefixIndexExprs()) {
+                if (index.getDataverseName().equals(MetadataConstants.METADATA_DATAVERSE_NAME)
+                        && !index.isPrimaryIndex()) {
+                    indexExprAndVarIt.remove();
+                    continue;
+                }
+            }
             for (int i = 0; i < index.getKeyFieldNames().size(); i++) {
                 List<String> keyField = index.getKeyFieldNames().get(i);
                 final IAType keyType = index.getKeyFieldTypes().get(i);
@@ -294,12 +304,6 @@ public abstract class AbstractIntroduceAccessMethodRule implements IAlgebraicRew
             }
             // A prefix of the index exprs may have been matched.
             if (accessMethod.matchPrefixIndexExprs()) {
-                // Remove the candidate if the dataset is a metadata dataset and the index is secondary
-                if (index.getDataverseName().equals(MetadataConstants.METADATA_DATAVERSE_NAME)
-                        && !index.isPrimaryIndex()) {
-                    indexExprAndVarIt.remove();
-                    continue;
-                }
                 if (lastFieldMatched < 0) {
                     indexExprAndVarIt.remove();
                     continue;

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/typecast/StaticTypeCastUtil.java
----------------------------------------------------------------------
diff --git a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/typecast/StaticTypeCastUtil.java b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/typecast/StaticTypeCastUtil.java
index 5ab27c6..f6bf3c3 100644
--- a/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/typecast/StaticTypeCastUtil.java
+++ b/asterix-algebra/src/main/java/edu/uci/ics/asterix/optimizer/rules/typecast/StaticTypeCastUtil.java
@@ -301,10 +301,8 @@ public class StaticTypeCastUtil {
                     }
 
                     // match the optional field
-                    if (reqFieldType.getTypeTag() == ATypeTag.UNION
-                            && NonTaggedFormatUtil.isOptionalField((AUnionType) reqFieldType)) {
-                        IAType itemType = ((AUnionType) reqFieldType).getUnionList().get(
-                                AUnionType.OPTIONAL_TYPE_INDEX_IN_UNION_LIST);
+                    if (NonTaggedFormatUtil.isOptional(reqFieldType)) {
+                        IAType itemType = ((AUnionType) reqFieldType).getNullableType();
                         reqFieldType = itemType;
                         if (fieldType.equals(BuiltinType.ANULL) || fieldType.equals(itemType)) {
                             fieldPermutation[j] = i;
@@ -322,10 +320,8 @@ public class StaticTypeCastUtil {
 
                     // match the optional type input for a non-optional field
                     // delay that to runtime by calling the not-null function
-                    if (fieldType.getTypeTag() == ATypeTag.UNION
-                            && NonTaggedFormatUtil.isOptionalField((AUnionType) fieldType)) {
-                        IAType itemType = ((AUnionType) fieldType).getUnionList().get(
-                                AUnionType.OPTIONAL_TYPE_INDEX_IN_UNION_LIST);
+                    if (NonTaggedFormatUtil.isOptional(fieldType)) {
+                        IAType itemType = ((AUnionType) fieldType).getNullableType();
                         if (reqFieldType.equals(itemType)) {
                             fieldPermutation[j] = i;
                             openFields[i] = false;
@@ -379,10 +375,8 @@ public class StaticTypeCastUtil {
                 }
 
                 // match the optional field
-                if (reqFieldType.getTypeTag() == ATypeTag.UNION
-                        && NonTaggedFormatUtil.isOptionalField((AUnionType) reqFieldType)) {
-                    IAType itemType = ((AUnionType) reqFieldType).getUnionList().get(
-                            AUnionType.OPTIONAL_TYPE_INDEX_IN_UNION_LIST);
+                if (NonTaggedFormatUtil.isOptional(reqFieldType)) {
+                    IAType itemType = ((AUnionType) reqFieldType).getNullableType();
                     if (fieldType.equals(BuiltinType.ANULL) || fieldType.equals(itemType)) {
                         matched = true;
                         break;
@@ -392,8 +386,7 @@ public class StaticTypeCastUtil {
             if (matched)
                 continue;
 
-            if (reqFieldType.getTypeTag() == ATypeTag.UNION
-                    && NonTaggedFormatUtil.isOptionalField((AUnionType) reqFieldType)) {
+            if (NonTaggedFormatUtil.isOptional(reqFieldType)) {
                 // add a null field
                 nullFields[i] = true;
             } else {

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-algebra/src/main/java/edu/uci/ics/asterix/translator/TypeTranslator.java
----------------------------------------------------------------------
diff --git a/asterix-algebra/src/main/java/edu/uci/ics/asterix/translator/TypeTranslator.java b/asterix-algebra/src/main/java/edu/uci/ics/asterix/translator/TypeTranslator.java
index d14b425..e6f3d17 100644
--- a/asterix-algebra/src/main/java/edu/uci/ics/asterix/translator/TypeTranslator.java
+++ b/asterix-algebra/src/main/java/edu/uci/ics/asterix/translator/TypeTranslator.java
@@ -40,6 +40,7 @@ import edu.uci.ics.asterix.om.types.ARecordType;
 import edu.uci.ics.asterix.om.types.AUnionType;
 import edu.uci.ics.asterix.om.types.AUnorderedListType;
 import edu.uci.ics.asterix.om.types.AbstractCollectionType;
+import edu.uci.ics.asterix.om.types.AbstractComplexType;
 import edu.uci.ics.asterix.om.types.BuiltinType;
 import edu.uci.ics.asterix.om.types.IAType;
 import edu.uci.ics.asterix.om.types.TypeSignature;
@@ -65,6 +66,9 @@ public class TypeTranslator {
         secondPass(mdTxnCtx, typeMap, incompleteFieldTypes, incompleteItemTypes, incompleteTopLevelTypeReferences,
                 typeDataverse);
 
+        for (IAType type : typeMap.values())
+            if (type.getTypeTag().isDerivedType())
+                ((AbstractComplexType) type).generateNestedDerivedTypeNames();
         return typeMap;
     }
 
@@ -341,12 +345,12 @@ public class TypeTranslator {
                         if (!rtd.getNullableFields().get(j)) { // not nullable
                             fldTypes[j] = tref;
                         } else { // nullable
-                            fldTypes[j] = makeUnionWithNull(null, tref);
+                            fldTypes[j] = AUnionType.createNullableType(tref);
                         }
                     } else {
                         addIncompleteFieldTypeReference(recType, j, tre, incompleteFieldTypes);
                         if (rtd.getNullableFields().get(j)) {
-                            fldTypes[j] = makeUnionWithNull(null, null);
+                            fldTypes[j] = AUnionType.createNullableType(null);
                         }
                     }
                     break;
@@ -358,7 +362,7 @@ public class TypeTranslator {
                     if (!rtd.getNullableFields().get(j)) { // not nullable
                         fldTypes[j] = t2;
                     } else { // nullable
-                        fldTypes[j] = makeUnionWithNull(null, t2);
+                        fldTypes[j] = AUnionType.createNullableType(t2);
                     }
                     break;
                 }
@@ -366,14 +370,14 @@ public class TypeTranslator {
                     OrderedListTypeDefinition oltd = (OrderedListTypeDefinition) texpr;
                     IAType t2 = computeOrderedListType(null, oltd, typeMap, incompleteItemTypes, incompleteFieldTypes,
                             defaultDataverse);
-                    fldTypes[j] = (rtd.getNullableFields().get(j)) ? makeUnionWithNull(null, t2) : t2;
+                    fldTypes[j] = (rtd.getNullableFields().get(j)) ? AUnionType.createNullableType(t2) : t2;
                     break;
                 }
                 case UNORDEREDLIST: {
                     UnorderedListTypeDefinition ultd = (UnorderedListTypeDefinition) texpr;
                     IAType t2 = computeUnorderedListType(null, ultd, typeMap, incompleteItemTypes,
                             incompleteFieldTypes, defaultDataverse);
-                    fldTypes[j] = (rtd.getNullableFields().get(j)) ? makeUnionWithNull(null, t2) : t2;
+                    fldTypes[j] = (rtd.getNullableFields().get(j)) ? AUnionType.createNullableType(t2) : t2;
                     break;
                 }
                 default: {
@@ -385,11 +389,4 @@ public class TypeTranslator {
 
         return recType;
     }
-
-    private static AUnionType makeUnionWithNull(String unionTypeName, IAType type) {
-        ArrayList<IAType> unionList = new ArrayList<IAType>(2);
-        unionList.add(BuiltinType.ANULL);
-        unionList.add(type);
-        return new AUnionType(unionList, unionTypeName);
-    }
 }

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/main/java/edu/uci/ics/asterix/aql/translator/AqlTranslator.java
----------------------------------------------------------------------
diff --git a/asterix-app/src/main/java/edu/uci/ics/asterix/aql/translator/AqlTranslator.java b/asterix-app/src/main/java/edu/uci/ics/asterix/aql/translator/AqlTranslator.java
index c34c297..5030c0d 100644
--- a/asterix-app/src/main/java/edu/uci/ics/asterix/aql/translator/AqlTranslator.java
+++ b/asterix-app/src/main/java/edu/uci/ics/asterix/aql/translator/AqlTranslator.java
@@ -517,10 +517,10 @@ public class AqlTranslator extends AbstractAqlTranslator {
         String datasetName = dd.getName().getValue();
         DatasetType dsType = dd.getDatasetType();
         String itemTypeName = dd.getItemTypeName().getValue();
-        Identifier ngNameId = dd.getDatasetDetailsDecl().getNodegroupName();
+        Identifier ngNameId = dd.getNodegroupName();
         String nodegroupName = getNodeGroupName(ngNameId, dd, dataverseName);
-        String compactionPolicy = dd.getDatasetDetailsDecl().getCompactionPolicy();
-        Map<String, String> compactionPolicyProperties = dd.getDatasetDetailsDecl().getCompactionPolicyProperties();
+        String compactionPolicy = dd.getCompactionPolicy();
+        Map<String, String> compactionPolicyProperties = dd.getCompactionPolicyProperties();
         boolean defaultCompactionPolicy = (compactionPolicy == null);
         boolean temp = dd.getDatasetDetailsDecl().isTemp();
 
@@ -549,6 +549,15 @@ public class AqlTranslator extends AbstractAqlTranslator {
             if (dt == null) {
                 throw new AlgebricksException(": type " + itemTypeName + " could not be found.");
             }
+            String ngName = ngNameId != null ? ngNameId.getValue() : configureNodegroupForDataset(dd, dataverseName,
+                    mdTxnCtx);
+
+            if (compactionPolicy == null) {
+                compactionPolicy = GlobalConfig.DEFAULT_COMPACTION_POLICY_NAME;
+                compactionPolicyProperties = GlobalConfig.DEFAULT_COMPACTION_POLICY_PROPERTIES;
+            } else {
+                validateCompactionPolicy(compactionPolicy, compactionPolicyProperties, mdTxnCtx, false);
+            }
             switch (dd.getDatasetType()) {
                 case INTERNAL: {
                     IAType itemType = dt.getDatatype();
@@ -562,45 +571,29 @@ public class AqlTranslator extends AbstractAqlTranslator {
                     List<IAType> partitioningTypes = aRecordType.validatePartitioningExpressions(partitioningExprs,
                             autogenerated);
 
-                    String ngName = ngNameId != null ? ngNameId.getValue() : configureNodegroupForDataset(dd,
-                            dataverseName, mdTxnCtx);
                     List<String> filterField = ((InternalDetailsDecl) dd.getDatasetDetailsDecl()).getFilterField();
+                    if (filterField != null) {
+                        aRecordType.validateFilterField(filterField);
+                    }
                     if (compactionPolicy == null) {
                         if (filterField != null) {
                             // If the dataset has a filter and the user didn't specify a merge policy, then we will pick the
                             // correlated-prefix as the default merge policy.
                             compactionPolicy = GlobalConfig.DEFAULT_FILTERED_DATASET_COMPACTION_POLICY_NAME;
                             compactionPolicyProperties = GlobalConfig.DEFAULT_COMPACTION_POLICY_PROPERTIES;
-                        } else {
-                            compactionPolicy = GlobalConfig.DEFAULT_COMPACTION_POLICY_NAME;
-                            compactionPolicyProperties = GlobalConfig.DEFAULT_COMPACTION_POLICY_PROPERTIES;
                         }
-                    } else {
-                        validateCompactionPolicy(compactionPolicy, compactionPolicyProperties, mdTxnCtx, false);
-                    }
-                    if (filterField != null) {
-                        aRecordType.validateFilterField(filterField);
                     }
                     datasetDetails = new InternalDatasetDetails(InternalDatasetDetails.FileStructure.BTREE,
                             InternalDatasetDetails.PartitioningStrategy.HASH, partitioningExprs, partitioningExprs,
-                            partitioningTypes, ngName, autogenerated, compactionPolicy, compactionPolicyProperties,
-                            filterField, temp);
+                            partitioningTypes, autogenerated, filterField, temp);
                     break;
                 }
                 case EXTERNAL: {
                     String adapter = ((ExternalDetailsDecl) dd.getDatasetDetailsDecl()).getAdapter();
                     Map<String, String> properties = ((ExternalDetailsDecl) dd.getDatasetDetailsDecl()).getProperties();
 
-                    String ngName = ngNameId != null ? ngNameId.getValue() : configureNodegroupForDataset(dd,
-                            dataverseName, mdTxnCtx);
-                    if (compactionPolicy == null) {
-                        compactionPolicy = GlobalConfig.DEFAULT_COMPACTION_POLICY_NAME;
-                        compactionPolicyProperties = GlobalConfig.DEFAULT_COMPACTION_POLICY_PROPERTIES;
-                    } else {
-                        validateCompactionPolicy(compactionPolicy, compactionPolicyProperties, mdTxnCtx, true);
-                    }
-                    datasetDetails = new ExternalDatasetDetails(adapter, properties, ngName, new Date(),
-                            ExternalDatasetTransactionState.COMMIT, compactionPolicy, compactionPolicyProperties);
+                    datasetDetails = new ExternalDatasetDetails(adapter, properties, new Date(),
+                            ExternalDatasetTransactionState.COMMIT);
                     break;
                 }
 
@@ -612,7 +605,8 @@ public class AqlTranslator extends AbstractAqlTranslator {
             }
 
             //#. add a new dataset with PendingAddOp
-            dataset = new Dataset(dataverseName, datasetName, itemTypeName, datasetDetails, dd.getHints(), dsType,
+            dataset = new Dataset(dataverseName, datasetName, itemTypeName, ngName, compactionPolicy,
+                    compactionPolicyProperties, datasetDetails, dd.getHints(), dsType,
                     DatasetIdFactory.generateDatasetId(), IMetadataEntity.PENDING_ADD_OP);
             MetadataManager.INSTANCE.addDataset(metadataProvider.getMetadataTxnContext(), dataset);
 
@@ -851,9 +845,6 @@ public class AqlTranslator extends AbstractAqlTranslator {
                 }
                 if (fieldType == null)
                     throw new AlgebricksException("Unknown type " + fieldExpr.second);
-                if (isOpen && fieldType.getTypeTag().isDerivedType())
-                    MetadataManager.INSTANCE.addDatatype(mdTxnCtx, new Datatype(dataverseName, indexName, fieldType,
-                            false));
 
                 indexFields.add(fieldExpr.first);
                 indexFieldTypes.add(fieldType);
@@ -1372,7 +1363,8 @@ public class AqlTranslator extends AbstractAqlTranslator {
                 MetadataManager.INSTANCE.dropDataset(mdTxnCtx, dataverseName, datasetName);
                 MetadataManager.INSTANCE.addDataset(
                         mdTxnCtx,
-                        new Dataset(dataverseName, datasetName, ds.getItemTypeName(), ds.getDatasetDetails(), ds
+                        new Dataset(dataverseName, datasetName, ds.getItemTypeName(), ds.getNodeGroupName(), ds
+                                .getCompactionPolicy(), ds.getCompactionPolicyProperties(), ds.getDatasetDetails(), ds
                                 .getHints(), ds.getDatasetType(), ds.getDatasetId(), IMetadataEntity.PENDING_DROP_OP));
 
                 MetadataManager.INSTANCE.commitTransaction(mdTxnCtx);
@@ -1414,7 +1406,8 @@ public class AqlTranslator extends AbstractAqlTranslator {
                 MetadataManager.INSTANCE.dropDataset(mdTxnCtx, dataverseName, datasetName);
                 MetadataManager.INSTANCE.addDataset(
                         mdTxnCtx,
-                        new Dataset(dataverseName, datasetName, ds.getItemTypeName(), ds.getDatasetDetails(), ds
+                        new Dataset(dataverseName, datasetName, ds.getItemTypeName(), ds.getNodeGroupName(), ds
+                                .getCompactionPolicy(), ds.getCompactionPolicyProperties(), ds.getDatasetDetails(), ds
                                 .getHints(), ds.getDatasetType(), ds.getDatasetId(), IMetadataEntity.PENDING_DROP_OP));
 
                 MetadataManager.INSTANCE.commitTransaction(mdTxnCtx);
@@ -1436,7 +1429,7 @@ public class AqlTranslator extends AbstractAqlTranslator {
             //#. finally, delete the dataset.
             MetadataManager.INSTANCE.dropDataset(mdTxnCtx, dataverseName, datasetName);
             // Drop the associated nodegroup
-            String nodegroup = ds.getDatasetDetails().getNodeGroupName();
+            String nodegroup = ds.getNodeGroupName();
             if (!nodegroup.equalsIgnoreCase(MetadataConstants.METADATA_DEFAULT_NODEGROUP_NAME)) {
                 MetadataManager.INSTANCE.dropNodegroup(mdTxnCtx, dataverseName + ":" + datasetName);
             }
@@ -1522,7 +1515,6 @@ public class AqlTranslator extends AbstractAqlTranslator {
                 }
             }
 
-
             if (ds.getDatasetType() == DatasetType.INTERNAL) {
                 indexName = stmtIndexDrop.getIndexName().getValue();
                 Index index = MetadataManager.INSTANCE.getIndex(mdTxnCtx, dataverseName, datasetName, indexName);
@@ -1952,7 +1944,7 @@ public class AqlTranslator extends AbstractAqlTranslator {
             MetadataLockManager.INSTANCE.createFeedEnd(dataverseName, dataverseName + "." + feedName);
         }
     }
-    
+
     private void handleCreateFeedPolicyStatement(AqlMetadataProvider metadataProvider, Statement stmt,
             IHyracksClientConnection hcc) throws Exception {
         MetadataTransactionContext mdTxnCtx = MetadataManager.INSTANCE.beginTransaction();
@@ -2050,7 +2042,6 @@ public class AqlTranslator extends AbstractAqlTranslator {
             }
             MetadataManager.INSTANCE.commitTransaction(mdTxnCtx);
 
-
         } catch (Exception e) {
             abort(e, e, mdTxnCtx);
             throw e;
@@ -2058,7 +2049,7 @@ public class AqlTranslator extends AbstractAqlTranslator {
             MetadataLockManager.INSTANCE.dropFeedEnd(dataverseName, dataverseName + "." + feedName);
         }
     }
-    
+
     private void handleDropFeedPolicyStatement(AqlMetadataProvider metadataProvider, Statement stmt,
             IHyracksClientConnection hcc) throws Exception {
 
@@ -2086,7 +2077,6 @@ public class AqlTranslator extends AbstractAqlTranslator {
         }
     }
 
-
     private void handleConnectFeedStatement(AqlMetadataProvider metadataProvider, Statement stmt,
             IHyracksClientConnection hcc) throws Exception {
 
@@ -2190,7 +2180,7 @@ public class AqlTranslator extends AbstractAqlTranslator {
             }
         }
     }
-    
+
     /**
      * Generates a subscription request corresponding to a connect feed request. In addition, provides a boolean
      * flag indicating if feed intake job needs to be started (source primary feed not found to be active).
@@ -2259,7 +2249,7 @@ public class AqlTranslator extends AbstractAqlTranslator {
         sourceFeedJoint.addConnectionRequest(request);
         return new Triple<FeedConnectionRequest, Boolean, List<IFeedJoint>>(request, needIntakeJob, jointsToRegister);
     }
-    
+
     /*
      * Gets the feed joint corresponding to the feed definition. Tuples constituting the feed are 
      * available at this feed joint.
@@ -2282,7 +2272,7 @@ public class AqlTranslator extends AbstractAqlTranslator {
 
         return new FeedJointKey(sourceFeed.getFeedId(), appliedFunctions);
     }
-    
+
     private void handleDisconnectFeedStatement(AqlMetadataProvider metadataProvider, Statement stmt,
             IHyracksClientConnection hcc) throws Exception {
         DisconnectFeedStatement cfs = (DisconnectFeedStatement) stmt;
@@ -2335,7 +2325,7 @@ public class AqlTranslator extends AbstractAqlTranslator {
                     dataverseName + "." + cfs.getFeedName());
         }
     }
-    
+
     private void handleSubscribeFeedStatement(AqlMetadataProvider metadataProvider, Statement stmt,
             IHyracksClientConnection hcc) throws Exception {
 
@@ -2367,7 +2357,6 @@ public class AqlTranslator extends AbstractAqlTranslator {
 
         try {
 
-          
             JobSpecification alteredJobSpec = FeedUtil.alterJobSpecificationForFeed(compiled, feedConnectionId, bfs
                     .getSubscriptionRequest().getPolicyParameters());
             MetadataManager.INSTANCE.commitTransaction(mdTxnCtx);
@@ -2889,13 +2878,13 @@ public class AqlTranslator extends AbstractAqlTranslator {
                         pregelixStmt.getDatasetNameTo(), true);
                 this.handleDatasetDropStatement(metadataProvider, dropStmt, hcc);
 
-                IDatasetDetailsDecl idd = new InternalDetailsDecl(new Identifier(toDataset.getDatasetDetails()
-                        .getNodeGroupName()), toIndex.getKeyFieldNames(), false, toDataset.getDatasetDetails()
-                        .getCompactionPolicy(), toDataset.getDatasetDetails().getCompactionPolicyProperties(), null,
-                        toDataset.getDatasetDetails().isTemp());
+                IDatasetDetailsDecl idd = new InternalDetailsDecl(toIndex.getKeyFieldNames(), false, null, toDataset
+                        .getDatasetDetails().isTemp());
                 DatasetDecl createToDataset = new DatasetDecl(new Identifier(dataverseNameTo),
-                        pregelixStmt.getDatasetNameTo(), new Identifier(toDataset.getItemTypeName()),
-                        toDataset.getHints(), toDataset.getDatasetType(), idd, false);
+                        pregelixStmt.getDatasetNameTo(), new Identifier(toDataset.getItemTypeName()), new Identifier(
+                                toDataset.getNodeGroupName()), toDataset.getCompactionPolicy(),
+                        toDataset.getCompactionPolicyProperties(), toDataset.getHints(), toDataset.getDatasetType(),
+                        idd, false);
                 this.handleCreateDatasetStatement(metadataProvider, createToDataset, hcc);
             } catch (Exception e) {
                 e.printStackTrace();

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/main/java/edu/uci/ics/asterix/file/ExternalIndexingOperations.java
----------------------------------------------------------------------
diff --git a/asterix-app/src/main/java/edu/uci/ics/asterix/file/ExternalIndexingOperations.java b/asterix-app/src/main/java/edu/uci/ics/asterix/file/ExternalIndexingOperations.java
index a519eaf..8f555cc 100644
--- a/asterix-app/src/main/java/edu/uci/ics/asterix/file/ExternalIndexingOperations.java
+++ b/asterix-app/src/main/java/edu/uci/ics/asterix/file/ExternalIndexingOperations.java
@@ -390,11 +390,11 @@ public class ExternalIndexingOperations {
     public static Dataset createTransactionDataset(Dataset dataset) {
         ExternalDatasetDetails originalDsd = (ExternalDatasetDetails) dataset.getDatasetDetails();
         ExternalDatasetDetails dsd = new ExternalDatasetDetails(originalDsd.getAdapter(), originalDsd.getProperties(),
-                originalDsd.getNodeGroupName(), originalDsd.getTimestamp(), ExternalDatasetTransactionState.BEGIN,
-                originalDsd.getCompactionPolicy(), originalDsd.getCompactionPolicyProperties());
+                originalDsd.getTimestamp(), ExternalDatasetTransactionState.BEGIN);
         Dataset transactionDatset = new Dataset(dataset.getDataverseName(), dataset.getDatasetName(),
-                dataset.getItemTypeName(), dsd, dataset.getHints(), DatasetType.EXTERNAL, dataset.getDatasetId(),
-                dataset.getPendingOp());
+                dataset.getItemTypeName(), dataset.getNodeGroupName(), dataset.getCompactionPolicy(),
+                dataset.getCompactionPolicyProperties(), dsd, dataset.getHints(), DatasetType.EXTERNAL,
+                dataset.getDatasetId(), dataset.getPendingOp());
         return transactionDatset;
     }
 

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type/meta03_builtin_type.1.ddl.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type/meta03_builtin_type.1.ddl.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type/meta03_builtin_type.1.ddl.aql
new file mode 100644
index 0000000..edeaebf
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type/meta03_builtin_type.1.ddl.aql
@@ -0,0 +1,13 @@
+/*
+ * Description  : Create closed type &  query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+drop dataverse testdv if exists;
+create dataverse testdv;
+
+create type testdv.testtype as closed {
+id : int32
+}
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type/meta03_builtin_type.2.update.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type/meta03_builtin_type.2.update.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type/meta03_builtin_type.2.update.aql
new file mode 100644
index 0000000..125f63b
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type/meta03_builtin_type.2.update.aql
@@ -0,0 +1,7 @@
+/*
+ * Description  : Create closed type &  query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type/meta03_builtin_type.3.query.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type/meta03_builtin_type.3.query.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type/meta03_builtin_type.3.query.aql
new file mode 100644
index 0000000..9861b7f
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type/meta03_builtin_type.3.query.aql
@@ -0,0 +1,11 @@
+/*
+ * Description  : Create closed type &  query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+
+for $l in dataset('Metadata.Datatype')
+where $l.DataverseName='testdv' and $l.DatatypeName='testtype'
+return $l
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type_nullable/meta03_builtin_type_nullable.1.ddl.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type_nullable/meta03_builtin_type_nullable.1.ddl.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type_nullable/meta03_builtin_type_nullable.1.ddl.aql
new file mode 100644
index 0000000..53e9032
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type_nullable/meta03_builtin_type_nullable.1.ddl.aql
@@ -0,0 +1,12 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+drop dataverse testdv if exists;
+create dataverse testdv;
+
+create type testdv.testtype as closed {
+id : int32?
+}

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type_nullable/meta03_builtin_type_nullable.2.update.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type_nullable/meta03_builtin_type_nullable.2.update.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type_nullable/meta03_builtin_type_nullable.2.update.aql
new file mode 100644
index 0000000..14bdd4d
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type_nullable/meta03_builtin_type_nullable.2.update.aql
@@ -0,0 +1,6 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type_nullable/meta03_builtin_type_nullable.3.query.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type_nullable/meta03_builtin_type_nullable.3.query.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type_nullable/meta03_builtin_type_nullable.3.query.aql
new file mode 100644
index 0000000..86c7695
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/builtin_type_nullable/meta03_builtin_type_nullable.3.query.aql
@@ -0,0 +1,11 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+
+for $l in dataset('Metadata.Datatype')
+where $l.DataverseName='testdv' and $l.DatatypeName='testtype'
+return $l
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list/meta03_ordered_list.1.ddl.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list/meta03_ordered_list.1.ddl.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list/meta03_ordered_list.1.ddl.aql
new file mode 100644
index 0000000..84d4ecc
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list/meta03_ordered_list.1.ddl.aql
@@ -0,0 +1,13 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+drop dataverse testdv if exists;
+create dataverse testdv;
+
+create type testdv.testtype as closed {
+id : int32,
+list: [ string ]
+}

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list/meta03_ordered_list.2.update.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list/meta03_ordered_list.2.update.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list/meta03_ordered_list.2.update.aql
new file mode 100644
index 0000000..14bdd4d
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list/meta03_ordered_list.2.update.aql
@@ -0,0 +1,6 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list/meta03_ordered_list.3.query.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list/meta03_ordered_list.3.query.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list/meta03_ordered_list.3.query.aql
new file mode 100644
index 0000000..86c7695
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list/meta03_ordered_list.3.query.aql
@@ -0,0 +1,11 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+
+for $l in dataset('Metadata.Datatype')
+where $l.DataverseName='testdv' and $l.DatatypeName='testtype'
+return $l
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list_nullable/meta03_ordered_list_nullable.1.ddl.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list_nullable/meta03_ordered_list_nullable.1.ddl.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list_nullable/meta03_ordered_list_nullable.1.ddl.aql
new file mode 100644
index 0000000..c3a8cc5
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list_nullable/meta03_ordered_list_nullable.1.ddl.aql
@@ -0,0 +1,13 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+drop dataverse testdv if exists;
+create dataverse testdv;
+
+create type testdv.testtype as closed {
+id : int32,
+list: [ string ]?
+}

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list_nullable/meta03_ordered_list_nullable.2.update.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list_nullable/meta03_ordered_list_nullable.2.update.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list_nullable/meta03_ordered_list_nullable.2.update.aql
new file mode 100644
index 0000000..14bdd4d
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list_nullable/meta03_ordered_list_nullable.2.update.aql
@@ -0,0 +1,6 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list_nullable/meta03_ordered_list_nullable.3.query.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list_nullable/meta03_ordered_list_nullable.3.query.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list_nullable/meta03_ordered_list_nullable.3.query.aql
new file mode 100644
index 0000000..86c7695
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/ordered_list_nullable/meta03_ordered_list_nullable.3.query.aql
@@ -0,0 +1,11 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+
+for $l in dataset('Metadata.Datatype')
+where $l.DataverseName='testdv' and $l.DatatypeName='testtype'
+return $l
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record/meta03_record.1.ddl.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record/meta03_record.1.ddl.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record/meta03_record.1.ddl.aql
new file mode 100644
index 0000000..764cfb7
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record/meta03_record.1.ddl.aql
@@ -0,0 +1,13 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+drop dataverse testdv if exists;
+create dataverse testdv;
+
+create type testdv.testtype as closed {
+id : int32,
+subtype: { sec_id: int32}
+}

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record/meta03_record.2.update.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record/meta03_record.2.update.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record/meta03_record.2.update.aql
new file mode 100644
index 0000000..14bdd4d
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record/meta03_record.2.update.aql
@@ -0,0 +1,6 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record/meta03_record.3.query.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record/meta03_record.3.query.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record/meta03_record.3.query.aql
new file mode 100644
index 0000000..86c7695
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record/meta03_record.3.query.aql
@@ -0,0 +1,11 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+
+for $l in dataset('Metadata.Datatype')
+where $l.DataverseName='testdv' and $l.DatatypeName='testtype'
+return $l
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record_nullable/meta03_record_nullable.1.ddl.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record_nullable/meta03_record_nullable.1.ddl.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record_nullable/meta03_record_nullable.1.ddl.aql
new file mode 100644
index 0000000..afc44ab
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record_nullable/meta03_record_nullable.1.ddl.aql
@@ -0,0 +1,13 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+drop dataverse testdv if exists;
+create dataverse testdv;
+
+create type testdv.testtype as closed {
+id : int32,
+subtype: { sec_id: int32}?
+}

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record_nullable/meta03_record_nullable.2.update.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record_nullable/meta03_record_nullable.2.update.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record_nullable/meta03_record_nullable.2.update.aql
new file mode 100644
index 0000000..14bdd4d
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record_nullable/meta03_record_nullable.2.update.aql
@@ -0,0 +1,6 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record_nullable/meta03_record_nullable.3.query.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record_nullable/meta03_record_nullable.3.query.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record_nullable/meta03_record_nullable.3.query.aql
new file mode 100644
index 0000000..86c7695
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/record_nullable/meta03_record_nullable.3.query.aql
@@ -0,0 +1,11 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+
+for $l in dataset('Metadata.Datatype')
+where $l.DataverseName='testdv' and $l.DatatypeName='testtype'
+return $l
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list/meta04_unordered_list.1.ddl.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list/meta04_unordered_list.1.ddl.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list/meta04_unordered_list.1.ddl.aql
new file mode 100644
index 0000000..0abb0e0
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list/meta04_unordered_list.1.ddl.aql
@@ -0,0 +1,13 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+drop dataverse testdv if exists;
+create dataverse testdv;
+
+create type testdv.testtype as closed {
+id : int32,
+list: {{ string }}
+}

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list/meta04_unordered_list.2.update.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list/meta04_unordered_list.2.update.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list/meta04_unordered_list.2.update.aql
new file mode 100644
index 0000000..14bdd4d
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list/meta04_unordered_list.2.update.aql
@@ -0,0 +1,6 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list/meta04_unordered_list.3.query.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list/meta04_unordered_list.3.query.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list/meta04_unordered_list.3.query.aql
new file mode 100644
index 0000000..86c7695
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list/meta04_unordered_list.3.query.aql
@@ -0,0 +1,11 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+
+for $l in dataset('Metadata.Datatype')
+where $l.DataverseName='testdv' and $l.DatatypeName='testtype'
+return $l
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list_nullable/meta04_unordered_list_nullable.1.ddl.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list_nullable/meta04_unordered_list_nullable.1.ddl.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list_nullable/meta04_unordered_list_nullable.1.ddl.aql
new file mode 100644
index 0000000..e1c5f81
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list_nullable/meta04_unordered_list_nullable.1.ddl.aql
@@ -0,0 +1,13 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+drop dataverse testdv if exists;
+create dataverse testdv;
+
+create type testdv.testtype as closed {
+id : int32,
+list: {{ string }}?
+}

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list_nullable/meta04_unordered_list_nullable.2.update.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list_nullable/meta04_unordered_list_nullable.2.update.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list_nullable/meta04_unordered_list_nullable.2.update.aql
new file mode 100644
index 0000000..14bdd4d
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list_nullable/meta04_unordered_list_nullable.2.update.aql
@@ -0,0 +1,6 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list_nullable/meta04_unordered_list_nullable.3.query.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list_nullable/meta04_unordered_list_nullable.3.query.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list_nullable/meta04_unordered_list_nullable.3.query.aql
new file mode 100644
index 0000000..86c7695
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta03/complex_type/unordered_list_nullable/meta04_unordered_list_nullable.3.query.aql
@@ -0,0 +1,11 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+
+for $l in dataset('Metadata.Datatype')
+where $l.DataverseName='testdv' and $l.DatatypeName='testtype'
+return $l
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/meta03.1.ddl.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/meta03.1.ddl.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/meta03.1.ddl.aql
deleted file mode 100644
index edeaebf..0000000
--- a/asterix-app/src/test/resources/metadata/queries/basic/meta03/meta03.1.ddl.aql
+++ /dev/null
@@ -1,13 +0,0 @@
-/*
- * Description  : Create closed type &  query Metadata dataset Datatype to verify.
- * Expected Res : Success
- * Date         : 15 Sep 2012
- */
-
-drop dataverse testdv if exists;
-create dataverse testdv;
-
-create type testdv.testtype as closed {
-id : int32
-}
-

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/meta03.2.update.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/meta03.2.update.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/meta03.2.update.aql
deleted file mode 100644
index 125f63b..0000000
--- a/asterix-app/src/test/resources/metadata/queries/basic/meta03/meta03.2.update.aql
+++ /dev/null
@@ -1,7 +0,0 @@
-/*
- * Description  : Create closed type &  query Metadata dataset Datatype to verify.
- * Expected Res : Success
- * Date         : 15 Sep 2012
- */
-
-

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta03/meta03.3.query.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta03/meta03.3.query.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta03/meta03.3.query.aql
deleted file mode 100644
index 9861b7f..0000000
--- a/asterix-app/src/test/resources/metadata/queries/basic/meta03/meta03.3.query.aql
+++ /dev/null
@@ -1,11 +0,0 @@
-/*
- * Description  : Create closed type &  query Metadata dataset Datatype to verify.
- * Expected Res : Success
- * Date         : 15 Sep 2012
- */
-
-
-for $l in dataset('Metadata.Datatype')
-where $l.DataverseName='testdv' and $l.DatatypeName='testtype'
-return $l
-

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type/meta04_builtin_type.1.ddl.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type/meta04_builtin_type.1.ddl.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type/meta04_builtin_type.1.ddl.aql
new file mode 100644
index 0000000..77c2793
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type/meta04_builtin_type.1.ddl.aql
@@ -0,0 +1,12 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+drop dataverse testdv if exists;
+create dataverse testdv;
+
+create type testdv.testtype as open {
+id : int32
+}

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type/meta04_builtin_type.2.update.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type/meta04_builtin_type.2.update.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type/meta04_builtin_type.2.update.aql
new file mode 100644
index 0000000..14bdd4d
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type/meta04_builtin_type.2.update.aql
@@ -0,0 +1,6 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type/meta04_builtin_type.3.query.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type/meta04_builtin_type.3.query.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type/meta04_builtin_type.3.query.aql
new file mode 100644
index 0000000..86c7695
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type/meta04_builtin_type.3.query.aql
@@ -0,0 +1,11 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+
+for $l in dataset('Metadata.Datatype')
+where $l.DataverseName='testdv' and $l.DatatypeName='testtype'
+return $l
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type_nullable/meta04_builtin_type_nullable.1.ddl.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type_nullable/meta04_builtin_type_nullable.1.ddl.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type_nullable/meta04_builtin_type_nullable.1.ddl.aql
new file mode 100644
index 0000000..48a2a84
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type_nullable/meta04_builtin_type_nullable.1.ddl.aql
@@ -0,0 +1,12 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+drop dataverse testdv if exists;
+create dataverse testdv;
+
+create type testdv.testtype as open {
+id : int32?
+}

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type_nullable/meta04_builtin_type_nullable.2.update.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type_nullable/meta04_builtin_type_nullable.2.update.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type_nullable/meta04_builtin_type_nullable.2.update.aql
new file mode 100644
index 0000000..14bdd4d
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type_nullable/meta04_builtin_type_nullable.2.update.aql
@@ -0,0 +1,6 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type_nullable/meta04_builtin_type_nullable.3.query.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type_nullable/meta04_builtin_type_nullable.3.query.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type_nullable/meta04_builtin_type_nullable.3.query.aql
new file mode 100644
index 0000000..86c7695
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta04/builtin_type_nullable/meta04_builtin_type_nullable.3.query.aql
@@ -0,0 +1,11 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+
+for $l in dataset('Metadata.Datatype')
+where $l.DataverseName='testdv' and $l.DatatypeName='testtype'
+return $l
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list/meta04_ordered_list.1.ddl.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list/meta04_ordered_list.1.ddl.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list/meta04_ordered_list.1.ddl.aql
new file mode 100644
index 0000000..4aa8067
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list/meta04_ordered_list.1.ddl.aql
@@ -0,0 +1,13 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+drop dataverse testdv if exists;
+create dataverse testdv;
+
+create type testdv.testtype as open {
+id : int32,
+list: [ string ]
+}

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list/meta04_ordered_list.2.update.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list/meta04_ordered_list.2.update.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list/meta04_ordered_list.2.update.aql
new file mode 100644
index 0000000..14bdd4d
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list/meta04_ordered_list.2.update.aql
@@ -0,0 +1,6 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list/meta04_ordered_list.3.query.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list/meta04_ordered_list.3.query.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list/meta04_ordered_list.3.query.aql
new file mode 100644
index 0000000..86c7695
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list/meta04_ordered_list.3.query.aql
@@ -0,0 +1,11 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+
+for $l in dataset('Metadata.Datatype')
+where $l.DataverseName='testdv' and $l.DatatypeName='testtype'
+return $l
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list_nullable/meta04_ordered_list_nullable.1.ddl.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list_nullable/meta04_ordered_list_nullable.1.ddl.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list_nullable/meta04_ordered_list_nullable.1.ddl.aql
new file mode 100644
index 0000000..f25f930
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list_nullable/meta04_ordered_list_nullable.1.ddl.aql
@@ -0,0 +1,13 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+drop dataverse testdv if exists;
+create dataverse testdv;
+
+create type testdv.testtype as open {
+id : int32,
+list: [ string ]?
+}

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list_nullable/meta04_ordered_list_nullable.2.update.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list_nullable/meta04_ordered_list_nullable.2.update.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list_nullable/meta04_ordered_list_nullable.2.update.aql
new file mode 100644
index 0000000..14bdd4d
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list_nullable/meta04_ordered_list_nullable.2.update.aql
@@ -0,0 +1,6 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/a5895308/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list_nullable/meta04_ordered_list_nullable.3.query.aql
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list_nullable/meta04_ordered_list_nullable.3.query.aql b/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list_nullable/meta04_ordered_list_nullable.3.query.aql
new file mode 100644
index 0000000..86c7695
--- /dev/null
+++ b/asterix-app/src/test/resources/metadata/queries/basic/meta04/complex_type/ordered_list_nullable/meta04_ordered_list_nullable.3.query.aql
@@ -0,0 +1,11 @@
+/*
+ * Description  : Create open type & query Metadata dataset Datatype to verify.
+ * Expected Res : Success
+ * Date         : 15 Sep 2012
+ */
+
+
+for $l in dataset('Metadata.Datatype')
+where $l.DataverseName='testdv' and $l.DatatypeName='testtype'
+return $l
+



Mime
View raw message