Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B7D402009F4 for ; Thu, 26 May 2016 19:09:05 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id B688E160A2C; Thu, 26 May 2016 17:09:05 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 8ACCA160A17 for ; Thu, 26 May 2016 19:09:04 +0200 (CEST) Received: (qmail 41811 invoked by uid 500); 26 May 2016 17:09:03 -0000 Mailing-List: contact commits-help@asterixdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.apache.org Delivered-To: mailing list commits@asterixdb.apache.org Received: (qmail 41802 invoked by uid 99); 26 May 2016 17:09:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2016 17:09:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 5E4F9CEA26 for ; Thu, 26 May 2016 17:09:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.646 X-Spam-Level: X-Spam-Status: No, score=-4.646 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.426] autolearn=disabled Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id mgwnbJNS1Ygy for ; Thu, 26 May 2016 17:09:01 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with SMTP id 3216E5F235 for ; Thu, 26 May 2016 17:09:00 +0000 (UTC) Received: (qmail 41775 invoked by uid 99); 26 May 2016 17:08:59 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2016 17:08:59 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 2CD7BDFC6D; Thu, 26 May 2016 17:08:59 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: mhubail@apache.org To: commits@asterixdb.incubator.apache.org Message-Id: <254b1460dbc04b7b8d1936db57c72e09@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: incubator-asterixdb git commit: ASTERIXDB-1419: Fix type checking for CollectionType Date: Thu, 26 May 2016 17:08:59 +0000 (UTC) archived-at: Thu, 26 May 2016 17:09:05 -0000 Repository: incubator-asterixdb Updated Branches: refs/heads/master 35390c82b -> bd1d98ce4 ASTERIXDB-1419: Fix type checking for CollectionType Change-Id: Ibf11d6c59ae00fe6d21fed8d75f199ee4ac9852c Reviewed-on: https://asterix-gerrit.ics.uci.edu/886 Reviewed-by: Jenkins Tested-by: Jenkins Reviewed-by: Yingyi Bu Project: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/commit/bd1d98ce Tree: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/tree/bd1d98ce Diff: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/diff/bd1d98ce Branch: refs/heads/master Commit: bd1d98ce49ae6b7b2272e2025c07d15ed6d37376 Parents: 35390c8 Author: Murtadha Hubail Authored: Wed May 25 15:02:21 2016 -0700 Committer: Murtadha Hubail Committed: Thu May 26 10:08:44 2016 -0700 ---------------------------------------------------------------------- ...e_1419_drop_type_with_collection_1.1.ddl.aql | 39 +++++++++++++ ...e_1419_drop_type_with_collection_2.1.ddl.aql | 40 +++++++++++++ .../src/test/resources/metadata/testsuite.xml | 11 ++++ .../apache/asterix/metadata/MetadataNode.java | 59 +++++++++++++------- .../apache/asterix/om/types/ARecordType.java | 10 ++++ .../org/apache/asterix/om/types/AUnionType.java | 5 +- .../om/types/AbstractCollectionType.java | 4 ++ .../asterix/om/types/AbstractComplexType.java | 1 + 8 files changed, 148 insertions(+), 21 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/bd1d98ce/asterixdb/asterix-app/src/test/resources/metadata/queries/basic/issue_1419_drop_type_with_collection_1/issue_1419_drop_type_with_collection_1.1.ddl.aql ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/metadata/queries/basic/issue_1419_drop_type_with_collection_1/issue_1419_drop_type_with_collection_1.1.ddl.aql b/asterixdb/asterix-app/src/test/resources/metadata/queries/basic/issue_1419_drop_type_with_collection_1/issue_1419_drop_type_with_collection_1.1.ddl.aql new file mode 100644 index 0000000..22b4d02 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/metadata/queries/basic/issue_1419_drop_type_with_collection_1/issue_1419_drop_type_with_collection_1.1.ddl.aql @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +/* + * Description : drop a type containing a collection + * Expected Res : Success + * Date : 25 May 2016 + * Issue : 1419 + */ + +drop dataverse test if exists; +create dataverse test; +use dataverse test; + +create type typeSnapshot if not exists as open { + countyID: int32, + timeBin: interval, + tweetCount: int32, + retweetCount: int32, + users: [int64], + top50HashTags: [{string:int32}] +} + +drop type typeSnapshot; http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/bd1d98ce/asterixdb/asterix-app/src/test/resources/metadata/queries/basic/issue_1419_drop_type_with_collection_2/issue_1419_drop_type_with_collection_2.1.ddl.aql ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/metadata/queries/basic/issue_1419_drop_type_with_collection_2/issue_1419_drop_type_with_collection_2.1.ddl.aql b/asterixdb/asterix-app/src/test/resources/metadata/queries/basic/issue_1419_drop_type_with_collection_2/issue_1419_drop_type_with_collection_2.1.ddl.aql new file mode 100644 index 0000000..55691ee --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/metadata/queries/basic/issue_1419_drop_type_with_collection_2/issue_1419_drop_type_with_collection_2.1.ddl.aql @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +/* + * Description : Drop a used type + * Expected Res : Failure + * Date : 25 May 2016 + * Issue : 1419 + */ + +drop dataverse test if exists; +create dataverse test; +use dataverse test; + +create type subType as { + id:int32, + list:[string] +} + +create type superType as { + superid:int32, + superlist:[subType] +} + +drop type subType; http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/bd1d98ce/asterixdb/asterix-app/src/test/resources/metadata/testsuite.xml ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/metadata/testsuite.xml b/asterixdb/asterix-app/src/test/resources/metadata/testsuite.xml index 7a85d57..8f35293 100644 --- a/asterixdb/asterix-app/src/test/resources/metadata/testsuite.xml +++ b/asterixdb/asterix-app/src/test/resources/metadata/testsuite.xml @@ -323,6 +323,17 @@ temp_dataset + + + issue_1419_drop_type_with_collection + + + + + none + Error: Cannot drop type test.subType being used by type test.superType_superlist + + http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/bd1d98ce/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java index 54ec084..b82c9ee 100644 --- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java +++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java @@ -22,6 +22,7 @@ package org.apache.asterix.metadata; import java.io.IOException; import java.rmi.RemoteException; import java.util.ArrayList; +import java.util.Arrays; import java.util.List; import org.apache.asterix.common.api.IAsterixAppRuntimeContext; @@ -77,6 +78,9 @@ import org.apache.asterix.om.base.AInt32; import org.apache.asterix.om.base.AMutableString; import org.apache.asterix.om.base.AString; import org.apache.asterix.om.types.ARecordType; +import org.apache.asterix.om.types.ATypeTag; +import org.apache.asterix.om.types.AUnionType; +import org.apache.asterix.om.types.AbstractComplexType; import org.apache.asterix.om.types.BuiltinType; import org.apache.asterix.om.types.IAType; import org.apache.asterix.transaction.management.opcallbacks.SecondaryIndexModificationOperationCallback; @@ -716,7 +720,7 @@ public class MetadataNode implements IMetadataNode { } private void confirmDatatypeIsUnused(JobId jobId, String dataverseName, String datatypeName) - throws MetadataException { + throws MetadataException, RemoteException { confirmDatatypeIsUnusedByDatatypes(jobId, dataverseName, datatypeName); confirmDatatypeIsUnusedByDatasets(jobId, dataverseName, datatypeName); } @@ -734,33 +738,48 @@ public class MetadataNode implements IMetadataNode { } private void confirmDatatypeIsUnusedByDatatypes(JobId jobId, String dataverseName, String datatypeName) - throws MetadataException { + throws MetadataException, RemoteException { //If any datatype uses this type, throw an error //TODO: Currently this loads all types into memory. This will need to be fixed for large numbers of types + Datatype dataTypeToBeDropped = getDatatype(jobId, dataverseName, datatypeName); + assert dataTypeToBeDropped != null; + IAType typeToBeDropped = dataTypeToBeDropped.getDatatype(); List datatypes = getAllDatatypes(jobId); - for (Datatype type : datatypes) { - if (!type.getDataverseName().equals(dataverseName)) { + for (Datatype dataType : datatypes) { + //skip types in different dataverses as well as the type to be dropped itself + if (!dataType.getDataverseName().equals(dataverseName) + || dataType.getDatatype().getTypeName().equals(datatypeName)) { continue; } - ARecordType recType = (ARecordType) type.getDatatype(); - for (IAType subType : recType.getFieldTypes()) { - if (subType.getTypeName().equals(datatypeName)) { - throw new MetadataException("Cannot drop type " + dataverseName + "." + datatypeName - + " being used by type " + dataverseName + "." + recType.getTypeName()); - } + + AbstractComplexType recType = (AbstractComplexType) dataType.getDatatype(); + if (recType.containsType(typeToBeDropped)) { + throw new MetadataException("Cannot drop type " + dataverseName + "." + datatypeName + + " being used by type " + dataverseName + "." + recType.getTypeName()); } } } private List getNestedComplexDatatypeNamesForThisDatatype(JobId jobId, String dataverseName, - String datatypeName) throws Exception { + String datatypeName) throws MetadataException, RemoteException { //Return all field types that aren't builtin types Datatype parentType = getDatatype(jobId, dataverseName, datatypeName); - ARecordType recType = (ARecordType) parentType.getDatatype(); - List nestedTypes = new ArrayList(); - for (IAType subType : recType.getFieldTypes()) { - if (!(subType instanceof BuiltinType)) { - nestedTypes.add(subType.getTypeName()); + + List subTypes = null; + if (parentType.getDatatype().getTypeTag() == ATypeTag.RECORD) { + ARecordType recType = (ARecordType) parentType.getDatatype(); + subTypes = Arrays.asList(recType.getFieldTypes()); + } else if (parentType.getDatatype().getTypeTag() == ATypeTag.UNION) { + AUnionType recType = (AUnionType) parentType.getDatatype(); + subTypes = recType.getUnionList(); + } + + List nestedTypes = new ArrayList<>(); + if (subTypes != null) { + for (IAType subType : subTypes) { + if (!(subType instanceof BuiltinType)) { + nestedTypes.add(subType.getTypeName()); + } } } return nestedTypes; @@ -977,8 +996,10 @@ public class MetadataNode implements IMetadataNode { try { while (rangeCursor.hasNext()) { rangeCursor.next(); - sb.append(TupleUtils.printTuple(rangeCursor.getTuple(), new ISerializerDeserializer[] { - AqlSerializerDeserializerProvider.INSTANCE.getSerializerDeserializer(BuiltinType.ASTRING), + sb.append(TupleUtils.printTuple(rangeCursor.getTuple(), + new ISerializerDeserializer[] { + AqlSerializerDeserializerProvider.INSTANCE.getSerializerDeserializer( + BuiltinType.ASTRING), AqlSerializerDeserializerProvider.INSTANCE.getSerializerDeserializer(BuiltinType.ASTRING), AqlSerializerDeserializerProvider.INSTANCE .getSerializerDeserializer(BuiltinType.ASTRING) })); @@ -995,7 +1016,7 @@ public class MetadataNode implements IMetadataNode { private void searchIndex(JobId jobId, IMetadataIndex index, ITupleReference searchKey, IValueExtractor valueExtractor, List results) - throws MetadataException, IndexException, IOException { + throws MetadataException, IndexException, IOException { IBinaryComparatorFactory[] comparatorFactories = index.getKeyBinaryComparatorFactory(); String resourceName = index.getFile().toString(); IIndex indexInstance = datasetLifecycleManager.getIndex(resourceName); http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/bd1d98ce/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/ARecordType.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/ARecordType.java b/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/ARecordType.java index 163421d..3cbac09 100644 --- a/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/ARecordType.java +++ b/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/ARecordType.java @@ -310,4 +310,14 @@ public class ARecordType extends AbstractComplexType { } return typeList; } + + @Override + public boolean containsType(IAType type) { + for (IAType aType : fieldTypes) { + if (aType.getTypeName().equals(type.getTypeName())) { + return true; + } + } + return false; + } } http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/bd1d98ce/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnionType.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnionType.java b/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnionType.java index 3c29e7f..8e92b81 100644 --- a/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnionType.java +++ b/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnionType.java @@ -56,9 +56,10 @@ public class AUnionType extends AbstractComplexType { return isMissableType() || isNullableType(); } - private boolean containsType(IAType t) { + @Override + public boolean containsType(IAType type) { for (int index = 0; index < unionList.size(); ++index) { - if (unionList.get(index) != null && unionList.get(index).equals(t)) { + if (unionList.get(index) != null && unionList.get(index).equals(type)) { return true; } } http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/bd1d98ce/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AbstractCollectionType.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AbstractCollectionType.java b/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AbstractCollectionType.java index 6e2ffef..88a4f93 100644 --- a/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AbstractCollectionType.java +++ b/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AbstractCollectionType.java @@ -63,4 +63,8 @@ public abstract class AbstractCollectionType extends AbstractComplexType { } } + @Override + public boolean containsType(IAType type) { + return isTyped() && itemType.getTypeName().equals(type.getTypeName()); + } } http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/bd1d98ce/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AbstractComplexType.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AbstractComplexType.java b/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AbstractComplexType.java index 1f43e38..9eddd7f 100644 --- a/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AbstractComplexType.java +++ b/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AbstractComplexType.java @@ -45,4 +45,5 @@ public abstract class AbstractComplexType implements IAType { return this.deepEqual((IAObject) object); } + public abstract boolean containsType(IAType type); }