Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 99ACC200D64 for ; Tue, 12 Dec 2017 07:24:21 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 97FE0160C22; Tue, 12 Dec 2017 06:24:21 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B7831160C13 for ; Tue, 12 Dec 2017 07:24:20 +0100 (CET) Received: (qmail 26151 invoked by uid 500); 12 Dec 2017 06:24:19 -0000 Mailing-List: contact notifications-help@asterixdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.apache.org Delivered-To: mailing list notifications@asterixdb.apache.org Received: (qmail 26142 invoked by uid 99); 12 Dec 2017 06:24:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Dec 2017 06:24:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 6D7331A0958 for ; Tue, 12 Dec 2017 06:24:19 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.919 X-Spam-Level: X-Spam-Status: No, score=0.919 tagged_above=-999 required=6.31 tests=[SPF_FAIL=0.919] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id eyY4J-IOGOmq for ; Tue, 12 Dec 2017 06:24:17 +0000 (UTC) Received: from vitalstatistix.ics.uci.edu (vitalstatistix.ics.uci.edu [128.195.52.38]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 48B7A5FB3E for ; Tue, 12 Dec 2017 06:24:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by vitalstatistix.ics.uci.edu (Postfix) with ESMTP id E6EEB100755; Mon, 11 Dec 2017 22:24:16 -0800 (PST) Date: Mon, 11 Dec 2017 22:24:15 -0800 From: "abdullah alamoudi (Code Review)" To: Jenkins CC: Till Westmann , Ali Alsuliman , Dmitry Lychagin Reply-To: bamousaa@gmail.com X-Gerrit-MessageType: newpatchset Subject: Change in asterixdb[master]: [ASTERIXDB-2194][COMP] Introduce datasource functions X-Gerrit-Change-Id: Ibcf807ac713a21e8f4d59868525467386e801303 X-Gerrit-ChangeURL: X-Gerrit-Commit: 8859631b642047b0433cbdefd50e64a71478d13e In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.7 Message-Id: <20171212062416.E6EEB100755@vitalstatistix.ics.uci.edu> archived-at: Tue, 12 Dec 2017 06:24:21 -0000 Hello Anon. E. Moose #1000171, Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/2216 to look at the new patch set (#7). Change subject: [ASTERIXDB-2194][COMP] Introduce datasource functions ...................................................................... [ASTERIXDB-2194][COMP] Introduce datasource functions - user model changes: yes Some functions can be datasources - storage format changes: no - interface changes: yes - Add IDatasourceFunction: A function that is also a datasource - Add IFunctionToDataSourceTransformer: transform an unnest function into a datascan during compilation Details: - Currently, functions are location agnostic and are run on parameters that are either passed through them during compile time or runtime. - An exception to this is the dataset function which has an associated location constraints running on the nodes which host the dataset. - In this change, we introduce a general framework that allows creation of new functions similar to the dataset function. - Such functions are called datasource Functions. - A datasource function takes constant parameters and run on a set of partitions similar to the dataset function. - The first example of such functions is the DatasetResources function. - The DatasetResources function takes two parameters, a dataverse and a dataset. It is then run on all nodes and returns a set of dataset resources. - Test cases are added for this function. Change-Id: Ibcf807ac713a21e8f4d59868525467386e801303 --- M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/AnalysisUtil.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/MetaFunctionToMetaVariableRule.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/UnnestToDataScanRule.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroducePrimaryIndexForAggregationRule.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/DatasetResourcesDatasource.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/DatasetResourcesFunction.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/DatasetResourcesReader.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/DatasetResourcesRewriter.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/DatasetRewriter.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/FeedRewriter.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/FunctionReader.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/FunctionRewriter.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/StorageComponentsDatasource.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/StorageComponentsFunction.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/StorageComponentsReader.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/StorageComponentsRewriter.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/util/MetadataBuiltinFunctions.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/FeedOperations.java A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/dataset-resources/dataset-resources.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/dataset-resources/dataset-resources.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/dataset-resources/dataset-resources.3.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/dataset-resources/dataset-resources.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/dataset-resources/dataset-resources.5.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/dataset-resources/dataset-resources.3.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/dataset-resources/dataset-resources.4.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/dataset-resources/dataset-resources.5.adm M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/adapter/factory/GenericAdapterFactory.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/FeedConstants.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/api/IDatasourceFunction.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataBuiltinEntities.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/AbstractDatasourceFunction.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/DataSource.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/FunctionDataSource.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/FunctionDataSourceFactory.java D asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/MetadataBuiltinFunctions.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/DatasetUtil.java M asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/BuiltinFunctions.java A asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/IFunctionToDataSourceRewriter.java M asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/ARecordType.java 42 files changed, 1,634 insertions(+), 499 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/16/2216/7 -- To view, visit https://asterix-gerrit.ics.uci.edu/2216 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibcf807ac713a21e8f4d59868525467386e801303 Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: abdullah alamoudi Gerrit-Reviewer: Ali Alsuliman Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi