Return-Path: X-Original-To: apmail-asterixdb-notifications-archive@minotaur.apache.org Delivered-To: apmail-asterixdb-notifications-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BE34D1848C for ; Fri, 12 Feb 2016 07:11:17 +0000 (UTC) Received: (qmail 60674 invoked by uid 500); 12 Feb 2016 07:11:17 -0000 Delivered-To: apmail-asterixdb-notifications-archive@asterixdb.apache.org Received: (qmail 60644 invoked by uid 500); 12 Feb 2016 07:11:17 -0000 Mailing-List: contact notifications-help@asterixdb.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.incubator.apache.org Delivered-To: mailing list notifications@asterixdb.incubator.apache.org Received: (qmail 60634 invoked by uid 99); 12 Feb 2016 07:11:17 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Feb 2016 07:11:17 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 26B96C19C2 for ; Fri, 12 Feb 2016 07:11:17 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.919 X-Spam-Level: X-Spam-Status: No, score=0.919 tagged_above=-999 required=6.31 tests=[SPF_FAIL=0.919] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 4da4P2fHEYGQ for ; Fri, 12 Feb 2016 07:11:15 +0000 (UTC) Received: from unhygienix.ics.uci.edu (unhygienix.ics.uci.edu [128.195.14.130]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTP id D31B443F5A for ; Fri, 12 Feb 2016 07:11:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by unhygienix.ics.uci.edu (Postfix) with ESMTP id 9F2DE2419ED; Thu, 11 Feb 2016 23:06:53 -0800 (PST) Date: Thu, 11 Feb 2016 23:06:53 -0800 From: "Yingyi Bu (Code Review)" Message-ID: Reply-To: buyingyi@gmail.com X-Gerrit-MessageType: newchange Subject: Change in asterixdb[master]: Fix ASTERIXDB-1298 X-Gerrit-Change-Id: Ibb594e161d7236131a7a6a0a290e76071d4587f4 X-Gerrit-ChangeURL: X-Gerrit-Commit: 3abd91f46b14ffb1ab66100832ad4e15eaba166f MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.8.4 To: undisclosed-recipients:; Yingyi Bu has uploaded a new change for review. https://asterix-gerrit.ics.uci.edu/630 Change subject: Fix ASTERIXDB-1298 ...................................................................... Fix ASTERIXDB-1298 Handle the case that the original join operator in a subplan operator is a left-outer join such that no null-check variable needs to be added. Change-Id: Ibb594e161d7236131a7a6a0a290e76071d4587f4 --- M asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java A asterix-app/src/test/resources/optimizerts/queries/udfs/query-ASTERIXDB-1298.aql A asterix-app/src/test/resources/optimizerts/results/udfs/query-ASTERIXDB-1298.plan A asterix-app/src/test/resources/runtimets/queries/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.1.ddl.aql A asterix-app/src/test/resources/runtimets/queries/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.2.update.aql A asterix-app/src/test/resources/runtimets/queries/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.3.query.aql A asterix-app/src/test/resources/runtimets/results/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.1.adm M asterix-app/src/test/resources/runtimets/testsuite.xml 8 files changed, 223 insertions(+), 22 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/30/630/1 diff --git a/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java b/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java index d88b69f..9f03ffa 100644 --- a/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java +++ b/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineSubplanInputForNestedTupleSourceRule.java @@ -503,29 +503,34 @@ } groupbyOp.getInputs().add(new MutableObject(topJoinRef.getValue())); - // Adds a select operator into the nested plan for group-by to remove tuples with NULL on {@code assignVar}, i.e., - // subplan input tuples that are filtered out within a subplan. - List> nullCheckExprRefs = new ArrayList<>(); - for (LogicalVariable notNullVar : notNullVars) { - Mutable filterVarExpr = new MutableObject( - new VariableReferenceExpression(notNullVar)); - List> args = new ArrayList>(); - args.add(filterVarExpr); - List> argsForNotFunction = new ArrayList>(); - argsForNotFunction.add(new MutableObject(new ScalarFunctionCallExpression( - FunctionUtil.getFunctionInfo(AsterixBuiltinFunctions.IS_NULL), args))); - nullCheckExprRefs.add(new MutableObject(new ScalarFunctionCallExpression( - FunctionUtil.getFunctionInfo(AsterixBuiltinFunctions.NOT), argsForNotFunction))); + if (notNullVars.size() > 0) { + // Adds a select operator into the nested plan for group-by to remove tuples with NULL on {@code assignVar}, i.e., + // subplan input tuples that are filtered out within a subplan. + List> nullCheckExprRefs = new ArrayList<>(); + for (LogicalVariable notNullVar : notNullVars) { + Mutable filterVarExpr = new MutableObject( + new VariableReferenceExpression(notNullVar)); + List> args = new ArrayList>(); + args.add(filterVarExpr); + List> argsForNotFunction = new ArrayList>(); + argsForNotFunction.add(new MutableObject(new ScalarFunctionCallExpression( + FunctionUtil.getFunctionInfo(AsterixBuiltinFunctions.IS_NULL), args))); + nullCheckExprRefs.add(new MutableObject(new ScalarFunctionCallExpression( + FunctionUtil.getFunctionInfo(AsterixBuiltinFunctions.NOT), argsForNotFunction))); + } + Mutable selectExprRef = nullCheckExprRefs.size() > 1 + ? new MutableObject(new ScalarFunctionCallExpression( + FunctionUtil.getFunctionInfo(AsterixBuiltinFunctions.AND), nullCheckExprRefs)) + : nullCheckExprRefs.get(0); + SelectOperator selectOp = new SelectOperator(selectExprRef, false, null); + topJoinRef.setValue(selectOp); + selectOp.getInputs().add(new MutableObject( + new NestedTupleSourceOperator(new MutableObject(groupbyOp)))); + } else { + // The original join operator in the Subplan is a left-outer join. + // Therefore, no null-check variable is injected and no SelectOperator needs to be added. + topJoinRef.setValue(new NestedTupleSourceOperator(new MutableObject(groupbyOp))); } - Mutable selectExprRef = nullCheckExprRefs.size() > 1 - ? new MutableObject(new ScalarFunctionCallExpression( - FunctionUtil.getFunctionInfo(AsterixBuiltinFunctions.AND), nullCheckExprRefs)) - : nullCheckExprRefs.get(0); - SelectOperator selectOp = new SelectOperator(selectExprRef, false, null); - topJoinRef.setValue(selectOp); - selectOp.getInputs().add(new MutableObject( - new NestedTupleSourceOperator(new MutableObject(groupbyOp)))); - opRef.setValue(groupbyOp); OperatorManipulationUtil.computeTypeEnvironmentBottomUp(groupbyOp, context); diff --git a/asterix-app/src/test/resources/optimizerts/queries/udfs/query-ASTERIXDB-1298.aql b/asterix-app/src/test/resources/optimizerts/queries/udfs/query-ASTERIXDB-1298.aql new file mode 100644 index 0000000..775e714 --- /dev/null +++ b/asterix-app/src/test/resources/optimizerts/queries/udfs/query-ASTERIXDB-1298.aql @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +drop dataverse emergencyTest if exists; +create dataverse emergencyTest; +use dataverse emergencyTest; + + +drop dataverse channels if exists; +create dataverse channels; +use dataverse channels; + +create type userLocation as closed +{ userId: int, roomNumber: int } + +create dataset userLocations(userLocation) +primary key userId; + +create function currentOccupancy($room) +{ + let $list := for $location in dataset userLocations + where $location.roomNumber = $room + return $location.userId return $list +}; + +create type subscription as { "id":uuid, "param0":int } create dataset subscriptions(subscription) +primary key id autogenerated; + +for $sub in dataset subscriptions +for $result in currentOccupancy($sub.param0) +return $result; \ No newline at end of file diff --git a/asterix-app/src/test/resources/optimizerts/results/udfs/query-ASTERIXDB-1298.plan b/asterix-app/src/test/resources/optimizerts/results/udfs/query-ASTERIXDB-1298.plan new file mode 100644 index 0000000..c62d34e --- /dev/null +++ b/asterix-app/src/test/resources/optimizerts/results/udfs/query-ASTERIXDB-1298.plan @@ -0,0 +1,20 @@ +-- DISTRIBUTE_RESULT |PARTITIONED| + -- ONE_TO_ONE_EXCHANGE |PARTITIONED| + -- STREAM_PROJECT |PARTITIONED| + -- ONE_TO_ONE_EXCHANGE |PARTITIONED| + -- HYBRID_HASH_JOIN [$$5][$$17] |PARTITIONED| + -- HASH_PARTITION_EXCHANGE [$$5] |PARTITIONED| + -- STREAM_PROJECT |PARTITIONED| + -- ASSIGN |PARTITIONED| + -- STREAM_PROJECT |PARTITIONED| + -- ONE_TO_ONE_EXCHANGE |PARTITIONED| + -- DATASOURCE_SCAN |PARTITIONED| + -- ONE_TO_ONE_EXCHANGE |PARTITIONED| + -- EMPTY_TUPLE_SOURCE |PARTITIONED| + -- HASH_PARTITION_EXCHANGE [$$17] |PARTITIONED| + -- STREAM_PROJECT |PARTITIONED| + -- ASSIGN |PARTITIONED| + -- ONE_TO_ONE_EXCHANGE |PARTITIONED| + -- DATASOURCE_SCAN |PARTITIONED| + -- ONE_TO_ONE_EXCHANGE |PARTITIONED| + -- EMPTY_TUPLE_SOURCE |PARTITIONED| diff --git a/asterix-app/src/test/resources/runtimets/queries/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.1.ddl.aql b/asterix-app/src/test/resources/runtimets/queries/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.1.ddl.aql new file mode 100644 index 0000000..dfe0882 --- /dev/null +++ b/asterix-app/src/test/resources/runtimets/queries/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.1.ddl.aql @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + + +drop dataverse channels if exists; +create dataverse channels; +use dataverse channels; + +create type userLocation as closed +{ userId: int, roomNumber: int } + +create dataset userLocations(userLocation) +primary key userId; + +create function currentOccupancy($room) +{ + let $list := for $location in dataset userLocations + where $location.roomNumber = $room + return $location.userId return $list +}; + +create type subscription as { "id":uuid, "param0":int } create dataset subscriptions(subscription) +primary key id autogenerated; diff --git a/asterix-app/src/test/resources/runtimets/queries/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.2.update.aql b/asterix-app/src/test/resources/runtimets/queries/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.2.update.aql new file mode 100644 index 0000000..b715a6e --- /dev/null +++ b/asterix-app/src/test/resources/runtimets/queries/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.2.update.aql @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +use dataverse channels; + +upsert into dataset userLocations( +[ +{"userId":1, "roomNumber":123} +, +{"userId":2, "roomNumber":123} +, +{"userId":3, "roomNumber":123} +, +{"userId":4, "roomNumber":123} +, +{"userId":5, "roomNumber":350} +, +{"userId":6, "roomNumber":350} +, +{"userId":7, "roomNumber":350} +, +{"userId":8, "roomNumber":350} +, +{"userId":9, "roomNumber":350} +, +{"userId":10,"roomNumber":210} +, +{"userId":11,"roomNumber":210} +, +{"userId":12,"roomNumber":210} +, +{"userId":13,"roomNumber":210} +, +{"userId":14,"roomNumber":210} +] +); + +insert into dataset subscriptions( +{"param0":123} +); \ No newline at end of file diff --git a/asterix-app/src/test/resources/runtimets/queries/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.3.query.aql b/asterix-app/src/test/resources/runtimets/queries/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.3.query.aql new file mode 100644 index 0000000..04b7d28 --- /dev/null +++ b/asterix-app/src/test/resources/runtimets/queries/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.3.query.aql @@ -0,0 +1,25 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +use dataverse channels; + +for $sub in dataset subscriptions +for $result in currentOccupancy($sub.param0) +order by $result +return $result; diff --git a/asterix-app/src/test/resources/runtimets/results/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.1.adm b/asterix-app/src/test/resources/runtimets/results/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.1.adm new file mode 100644 index 0000000..94ebaf9 --- /dev/null +++ b/asterix-app/src/test/resources/runtimets/results/user-defined-functions/query-ASTERIXDB-1298/query-ASTERIXDB-1298.1.adm @@ -0,0 +1,4 @@ +1 +2 +3 +4 diff --git a/asterix-app/src/test/resources/runtimets/testsuite.xml b/asterix-app/src/test/resources/runtimets/testsuite.xml index 9b03eaf..5e83e1f 100644 --- a/asterix-app/src/test/resources/runtimets/testsuite.xml +++ b/asterix-app/src/test/resources/runtimets/testsuite.xml @@ -5946,6 +5946,11 @@ SyntaxError: function test.int8@0 is undefined + + + query-ASTERIXDB-1298 + +