Return-Path: X-Original-To: apmail-asterixdb-notifications-archive@minotaur.apache.org Delivered-To: apmail-asterixdb-notifications-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 14536182F7 for ; Tue, 24 Nov 2015 20:39:31 +0000 (UTC) Received: (qmail 23051 invoked by uid 500); 24 Nov 2015 20:39:31 -0000 Delivered-To: apmail-asterixdb-notifications-archive@asterixdb.apache.org Received: (qmail 23020 invoked by uid 500); 24 Nov 2015 20:39:31 -0000 Mailing-List: contact notifications-help@asterixdb.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.incubator.apache.org Delivered-To: mailing list notifications@asterixdb.incubator.apache.org Received: (qmail 23001 invoked by uid 99); 24 Nov 2015 20:39:30 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2015 20:39:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 5CA461A2EB6 for ; Tue, 24 Nov 2015 20:39:30 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.92 X-Spam-Level: X-Spam-Status: No, score=0.92 tagged_above=-999 required=6.31 tests=[SPF_FAIL=0.919, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id zyjhJLzgvHhJ for ; Tue, 24 Nov 2015 20:39:15 +0000 (UTC) Received: from unhygienix.ics.uci.edu (unhygienix.ics.uci.edu [128.195.14.130]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTP id B420422F01 for ; Tue, 24 Nov 2015 20:39:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by unhygienix.ics.uci.edu (Postfix) with ESMTP id 3246C240C34; Tue, 24 Nov 2015 12:36:03 -0800 (PST) Date: Tue, 24 Nov 2015 12:36:03 -0800 From: "Ian Maxon (Code Review)" To: abdullah alamoudi CC: Jenkins Reply-To: imaxon@apache.org X-Gerrit-MessageType: merged Subject: Change in asterixdb[release-0.8.8]: ASTERIXDB-1126 Correct comparators and type traits for exter... X-Gerrit-Change-Id: I71d924d7e2b7a7e6c752bc97679e612946afc17c X-Gerrit-ChangeURL: X-Gerrit-Commit: 5a78ca0f33354aab2163ab713a9fa98dca96d78d In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.8.4 Message-Id: <20151124203603.3246C240C34@unhygienix.ics.uci.edu> Ian Maxon has submitted this change and it was merged. Change subject: ASTERIXDB-1126 Correct comparators and type traits for external index ...................................................................... ASTERIXDB-1126 Correct comparators and type traits for external index In case of external data, the type definition does not contain information about primary keys and we should get them based on the data input format. Change-Id: I71d924d7e2b7a7e6c752bc97679e612946afc17c Reviewed-on: https://asterix-gerrit.ics.uci.edu/504 Tested-by: Jenkins Reviewed-by: Ian Maxon --- M asterix-app/data/csv/sample_01.csv M asterix-app/pom.xml M asterix-app/src/test/java/org/apache/asterix/test/runtime/HDFSCluster.java M asterix-installer/pom.xml M asterix-installer/src/test/java/org/apache/asterix/installer/transaction/RecoveryIT.java A asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.1.script.aql A asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.2.ddl.aql A asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.3.script.aql A asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.4.query.aql A asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.5.script.aql A asterix-installer/src/test/resources/transactionts/results/query_after_restart/external_index/external_index.1.adm A asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/external_index/create_and_start.sh A asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/external_index/stop_and_delete.sh A asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/external_index/stop_and_start.sh M asterix-installer/src/test/resources/transactionts/testsuite.xml M asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/AqlMetadataProvider.java 16 files changed, 147 insertions(+), 22 deletions(-) Approvals: Ian Maxon: Looks good to me, approved Jenkins: Verified diff --git a/asterix-app/data/csv/sample_01.csv b/asterix-app/data/csv/sample_01.csv index 4dd437a..fbba382 100644 --- a/asterix-app/data/csv/sample_01.csv +++ b/asterix-app/data/csv/sample_01.csv @@ -1,8 +1,8 @@ -1,0.899682764,5.6256,2013-08-07,07:22:35,1979-02-25T23:48:27.034 -2,0.669052398,,-1923-03-29,19:33:34,-1979-02-25T23:48:27.002 -3,0.572733058,192674,-1923-03-28,19:33:34,-1979-02-25T23:48:27.001 -4,,192674,-1923-03-27,19:33:34,-1979-02-25T23:48:27.001 -5,0.572733058,192674,,19:33:34,-1979-02-25T23:48:27.001 -6,0.572733058,192674,-1923-03-25,,-1979-02-25T23:48:27.001 -7,0.572733058,192674,-1923-03-24,19:33:34, +1,0.899682764,5.6256,2013-08-07,07:22:35,1979-02-25T23:48:27.034 +2,0.669052398,,-1923-03-29,19:33:34,-1979-02-25T23:48:27.002 +3,0.572733058,192674,-1923-03-28,19:33:34,-1979-02-25T23:48:27.001 +4,,192674,-1923-03-27,19:33:34,-1979-02-25T23:48:27.001 +5,0.572733058,192674,,19:33:34,-1979-02-25T23:48:27.001 +6,0.572733058,192674,-1923-03-25,,-1979-02-25T23:48:27.001 +7,0.572733058,192674,-1923-03-24,19:33:34, 8,,,,, \ No newline at end of file diff --git a/asterix-app/pom.xml b/asterix-app/pom.xml index c9a1b8e..85c9336 100644 --- a/asterix-app/pom.xml +++ b/asterix-app/pom.xml @@ -94,6 +94,18 @@ + + org.apache.maven.plugins + maven-jar-plugin + + + + jar + test-jar + + + + diff --git a/asterix-app/src/test/java/org/apache/asterix/test/runtime/HDFSCluster.java b/asterix-app/src/test/java/org/apache/asterix/test/runtime/HDFSCluster.java index 6d2de65..66680c4 100644 --- a/asterix-app/src/test/java/org/apache/asterix/test/runtime/HDFSCluster.java +++ b/asterix-app/src/test/java/org/apache/asterix/test/runtime/HDFSCluster.java @@ -36,7 +36,6 @@ * * @author ramangrover29 */ -@SuppressWarnings("deprecation") public class HDFSCluster { private static final String PATH_TO_HADOOP_CONF = "src/test/resources/hadoop/conf"; @@ -64,25 +63,27 @@ * Called prior to running the Runtime test suite. */ public void setup() throws Exception { - conf.addResource(new Path(PATH_TO_HADOOP_CONF + "/core-site.xml")); - conf.addResource(new Path(PATH_TO_HADOOP_CONF + "/mapred-site.xml")); - conf.addResource(new Path(PATH_TO_HADOOP_CONF + "/hdfs-site.xml")); + setup(""); + } + + public void setup(String basePath) throws Exception { + conf.addResource(new Path(basePath + PATH_TO_HADOOP_CONF + "/core-site.xml")); + conf.addResource(new Path(basePath + PATH_TO_HADOOP_CONF + "/mapred-site.xml")); + conf.addResource(new Path(basePath + PATH_TO_HADOOP_CONF + "/hdfs-site.xml")); cleanupLocal(); - //this constructor is deprecated in hadoop 2x - //dfsCluster = new MiniDFSCluster(nameNodePort, conf, numDataNodes, true, true, StartupOption.REGULAR, null); MiniDFSCluster.Builder build = new MiniDFSCluster.Builder(conf); build.nameNodePort(nameNodePort); build.numDataNodes(numDataNodes); build.startupOption(StartupOption.REGULAR); dfsCluster = build.build(); dfs = FileSystem.get(conf); - loadData(); + loadData(basePath); } - private void loadData() throws IOException { + private void loadData(String localDataRoot) throws IOException { Path destDir = new Path(HDFS_PATH); dfs.mkdirs(destDir); - File srcDir = new File(DATA_PATH); + File srcDir = new File(localDataRoot + DATA_PATH); File[] listOfFiles = srcDir.listFiles(); for (File srcFile : listOfFiles) { Path path = new Path(srcFile.getAbsolutePath()); @@ -108,12 +109,10 @@ HDFSCluster cluster = new HDFSCluster(); cluster.setup(); JobConf conf = configureJobConf(); - FileSystem fs = FileSystem.get(conf); InputSplit[] inputSplits = conf.getInputFormat().getSplits(conf, 0); for (InputSplit split : inputSplits) { System.out.println("split :" + split); } - // cluster.cleanup(); } private static JobConf configureJobConf() throws Exception { diff --git a/asterix-installer/pom.xml b/asterix-installer/pom.xml index 9a4857b..0be54d9 100644 --- a/asterix-installer/pom.xml +++ b/asterix-installer/pom.xml @@ -262,6 +262,13 @@ test-jar test + + org.apache.asterix + asterix-app + 0.8.8-SNAPSHOT + test-jar + test + org.apache.asterix asterix-server diff --git a/asterix-installer/src/test/java/org/apache/asterix/installer/transaction/RecoveryIT.java b/asterix-installer/src/test/java/org/apache/asterix/installer/transaction/RecoveryIT.java index c70b504..9e84e24 100644 --- a/asterix-installer/src/test/java/org/apache/asterix/installer/transaction/RecoveryIT.java +++ b/asterix-installer/src/test/java/org/apache/asterix/installer/transaction/RecoveryIT.java @@ -34,6 +34,7 @@ import org.junit.runners.Parameterized.Parameters; import org.apache.asterix.test.aql.TestsUtils; +import org.apache.asterix.test.runtime.HDFSCluster; import org.apache.asterix.testframework.context.TestCaseContext; @RunWith(Parameterized.class) @@ -42,6 +43,7 @@ private static final Logger LOGGER = Logger.getLogger(RecoveryIT.class.getName()); private static final String PATH_ACTUAL = "rttest/"; private static final String PATH_BASE = "src/test/resources/transactionts/"; + private static final String HDFS_BASE = "../asterix-app/"; private TestCaseContext tcCtx; private static File asterixInstallerPath; private static File installerTargetPath; @@ -79,6 +81,7 @@ + "configure_and_validate.sh"); TestsUtils.executeScript(pb, scriptHomePath + File.separator + "setup_teardown" + File.separator + "stop_and_delete.sh"); + HDFSCluster.getInstance().setup(HDFS_BASE); } @AfterClass @@ -92,6 +95,7 @@ + "stop_and_delete.sh"); TestsUtils.executeScript(pb, scriptHomePath + File.separator + "setup_teardown" + File.separator + "shutdown.sh"); + HDFSCluster.getInstance().cleanup(); } @Parameters diff --git a/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.1.script.aql b/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.1.script.aql new file mode 100644 index 0000000..cc46136 --- /dev/null +++ b/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.1.script.aql @@ -0,0 +1 @@ +create_and_start.sh diff --git a/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.2.ddl.aql b/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.2.ddl.aql new file mode 100644 index 0000000..7aa1129 --- /dev/null +++ b/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.2.ddl.aql @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +/* +* Description : Create an external dataset that contains records stored with text hdfs file format. + Build an index over the external dataset age attribute + Perform a query over the dataset using the index. +* Expected Res : Success +* Date : 3rd Jan 2014 +*/ +drop dataverse test if exists; +create dataverse test; + +use dataverse test; + +create type EmployeeType as closed { + id: int64, + name: string, + age: int64 +}; + +create external dataset EmployeeDataset(EmployeeType) +using hdfs +(("hdfs"="hdfs://127.0.0.1:31888"),("path"="/asterix/external-indexing-test.txt"),("input-format"="text-input-format"),("format"="delimited-text"),("delimiter"="|")); + +create index EmployeeAgeIdx on EmployeeDataset(age); + diff --git a/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.3.script.aql b/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.3.script.aql new file mode 100644 index 0000000..3ba1dc0 --- /dev/null +++ b/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.3.script.aql @@ -0,0 +1 @@ +stop_and_start.sh diff --git a/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.4.query.aql b/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.4.query.aql new file mode 100644 index 0000000..edfb290 --- /dev/null +++ b/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.4.query.aql @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +/* +* Description : Create an external dataset that contains records stored with text hdfs file format. + Build an index over the external dataset age attribute + Perform a query over the dataset using the index. +* Expected Res : Success +* Date : 3rd Jan 2014 +*/ +use dataverse test; + +for $emp in dataset EmployeeDataset +where $emp.age = 22 +order by $emp.id +return $emp; + diff --git a/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.5.script.aql b/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.5.script.aql new file mode 100644 index 0000000..10e1a51 --- /dev/null +++ b/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/external_index/external_index.5.script.aql @@ -0,0 +1 @@ +stop_and_delete.sh diff --git a/asterix-installer/src/test/resources/transactionts/results/query_after_restart/external_index/external_index.1.adm b/asterix-installer/src/test/resources/transactionts/results/query_after_restart/external_index/external_index.1.adm new file mode 100644 index 0000000..36b836c --- /dev/null +++ b/asterix-installer/src/test/resources/transactionts/results/query_after_restart/external_index/external_index.1.adm @@ -0,0 +1,4 @@ +[ { "id": 3, "name": "Samuel", "age": 22 } +, { "id": 10, "name": "David", "age": 22 } + ] + diff --git a/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/external_index/create_and_start.sh b/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/external_index/create_and_start.sh new file mode 100755 index 0000000..945f01d --- /dev/null +++ b/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/external_index/create_and_start.sh @@ -0,0 +1 @@ +$MANAGIX_HOME/bin/managix create -n nc1 -c $MANAGIX_HOME/clusters/local/local.xml; diff --git a/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/external_index/stop_and_delete.sh b/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/external_index/stop_and_delete.sh new file mode 100755 index 0000000..d7deea3 --- /dev/null +++ b/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/external_index/stop_and_delete.sh @@ -0,0 +1,3 @@ +$MANAGIX_HOME/bin/managix stop -n nc1; +$MANAGIX_HOME/bin/managix delete -n nc1; + diff --git a/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/external_index/stop_and_start.sh b/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/external_index/stop_and_start.sh new file mode 100755 index 0000000..1271a2b --- /dev/null +++ b/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/external_index/stop_and_start.sh @@ -0,0 +1,2 @@ +$MANAGIX_HOME/bin/managix stop -n nc1; +$MANAGIX_HOME/bin/managix start -n nc1; diff --git a/asterix-installer/src/test/resources/transactionts/testsuite.xml b/asterix-installer/src/test/resources/transactionts/testsuite.xml index cf95132..e3774f8 100644 --- a/asterix-installer/src/test/resources/transactionts/testsuite.xml +++ b/asterix-installer/src/test/resources/transactionts/testsuite.xml @@ -18,6 +18,13 @@ !--> + + + + external_index + + + diff --git a/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/AqlMetadataProvider.java b/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/AqlMetadataProvider.java index 2478dc7..e2fd888 100644 --- a/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/AqlMetadataProvider.java +++ b/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/AqlMetadataProvider.java @@ -708,7 +708,7 @@ } Pair comparatorFactoriesAndTypeTraits = getComparatorFactoriesAndTypeTraitsOfSecondaryBTreeIndex( secondaryIndex.getIndexType(), secondaryIndex.getKeyFieldNames(), - secondaryIndex.getKeyFieldTypes(), DatasetUtils.getPartitioningKeys(dataset), itemType); + secondaryIndex.getKeyFieldTypes(), DatasetUtils.getPartitioningKeys(dataset), itemType, dataset.getDatasetType()); comparatorFactories = comparatorFactoriesAndTypeTraits.first; typeTraits = comparatorFactoriesAndTypeTraits.second; if (filterTypeTraits != null) { @@ -810,7 +810,7 @@ private Pair getComparatorFactoriesAndTypeTraitsOfSecondaryBTreeIndex( IndexType indexType, List> sidxKeyFieldNames, List sidxKeyFieldTypes, - List> pidxKeyFieldNames, ARecordType recType) throws AlgebricksException { + List> pidxKeyFieldNames, ARecordType recType, DatasetType dsType) throws AlgebricksException { IBinaryComparatorFactory[] comparatorFactories; ITypeTraits[] typeTraits; @@ -832,8 +832,17 @@ for (int j = 0; j < pidxKeyFieldCount; ++j, ++i) { IAType keyType = null; try { - keyType = recType.getSubFieldType(pidxKeyFieldNames.get(j)); - } catch (IOException e) { + switch (dsType) { + case INTERNAL: + keyType = recType.getSubFieldType(pidxKeyFieldNames.get(j)); + break; + case EXTERNAL: + keyType = IndexingConstants.getFieldType(j); + break; + default: + throw new AlgebricksException("Unknown Dataset Type"); + } + } catch (IOException| AsterixException e) { throw new AlgebricksException(e); } comparatorFactories[i] = AqlBinaryComparatorFactoryProvider.INSTANCE.getBinaryComparatorFactory(keyType, -- To view, visit https://asterix-gerrit.ics.uci.edu/504 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: merged Gerrit-Change-Id: I71d924d7e2b7a7e6c752bc97679e612946afc17c Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: release-0.8.8 Gerrit-Owner: abdullah alamoudi Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: abdullah alamoudi