Return-Path: X-Original-To: apmail-asterixdb-commits-archive@minotaur.apache.org Delivered-To: apmail-asterixdb-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DA8F018E00 for ; Tue, 17 Nov 2015 00:55:39 +0000 (UTC) Received: (qmail 11375 invoked by uid 500); 17 Nov 2015 00:55:39 -0000 Delivered-To: apmail-asterixdb-commits-archive@asterixdb.apache.org Received: (qmail 11340 invoked by uid 500); 17 Nov 2015 00:55:39 -0000 Mailing-List: contact commits-help@asterixdb.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.incubator.apache.org Delivered-To: mailing list commits@asterixdb.incubator.apache.org Received: (qmail 11330 invoked by uid 99); 17 Nov 2015 00:55:39 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2015 00:55:39 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 2FCA81A094B for ; Tue, 17 Nov 2015 00:55:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.771 X-Spam-Level: * X-Spam-Status: No, score=1.771 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, T_RP_MATCHES_RCVD=-0.01, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id fG8Tzbjih90p for ; Tue, 17 Nov 2015 00:55:25 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with SMTP id 4992420C70 for ; Tue, 17 Nov 2015 00:55:23 +0000 (UTC) Received: (qmail 11238 invoked by uid 99); 17 Nov 2015 00:55:22 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2015 00:55:22 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id E2474E0A42; Tue, 17 Nov 2015 00:55:21 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: buyingyi@apache.org To: commits@asterixdb.incubator.apache.org Date: Tue, 17 Nov 2015 00:55:22 -0000 Message-Id: In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [2/2] incubator-asterixdb-hyracks git commit: SQL++ support in Algebricks: 1. added the OuterUnnestOperator; 2. fixed several rewriting rules. SQL++ support in Algebricks: 1. added the OuterUnnestOperator; 2. fixed several rewriting rules. Change-Id: I7dcf57f75ebc0a741b6ec9597525e226b6014fc0 Reviewed-on: https://asterix-gerrit.ics.uci.edu/314 Tested-by: Jenkins Reviewed-by: Till Westmann Project: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/commit/f9f918a1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/tree/f9f918a1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/diff/f9f918a1 Branch: refs/heads/master Commit: f9f918a11c8818adf76bb92b0c93a58cb3af2e1f Parents: 8af1963 Author: Yingyi Bu Authored: Sun Nov 15 22:28:05 2015 -0800 Committer: Yingyi Bu Committed: Mon Nov 16 16:51:42 2015 -0800 ---------------------------------------------------------------------- .../SequentialFixpointRuleController.java | 1 - .../core/algebra/base/LogicalOperatorTag.java | 1 + .../BroadcastExpressionAnnotation.java | 5 + .../IndexedNLJoinExpressionAnnotation.java | 5 + .../logical/AbstractLogicalOperator.java | 4 +- .../logical/AbstractUnnestNonMapOperator.java | 125 ++++++++++++++++++ .../operators/logical/OuterUnnestOperator.java | 71 +++++++++++ .../algebra/operators/logical/SinkOperator.java | 22 ++-- .../operators/logical/UnnestOperator.java | 95 +------------- .../visitors/FDsAndEquivClassesVisitor.java | 26 ++-- .../visitors/IsomorphismOperatorVisitor.java | 15 +++ .../IsomorphismVariableMappingVisitor.java | 7 ++ .../visitors/LogicalPropertiesVisitor.java | 17 ++- .../visitors/OperatorDeepCopyVisitor.java | 38 +++--- .../visitors/ProducedVariableVisitor.java | 25 ++-- .../logical/visitors/SchemaVariableVisitor.java | 7 ++ .../visitors/SubstituteVariableVisitor.java | 33 +++-- .../logical/visitors/UsedVariableVisitor.java | 7 ++ .../LogicalOperatorPrettyPrintVisitor.java | 32 +++-- .../properties/TypePropagationPolicy.java | 3 +- .../visitors/ILogicalOperatorVisitor.java | 3 + .../rewriter/base/AbstractRuleController.java | 3 +- .../rewriter/base/IAlgebraicRewriteRule.java | 1 - .../rules/IntroHashPartitionMergeExchange.java | 78 ------------ .../rewriter/rules/NestedSubplanToJoinRule.java | 30 +++-- .../rewriter/rules/PushSortDownRule.java | 97 ++++++++++++++ .../RemoveUnnecessarySortMergeExchange.java | 126 +++++++++++++++++++ .../rules/SimpleUnnestToProductRule.java | 6 +- 28 files changed, 625 insertions(+), 258 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-compiler/src/main/java/org/apache/hyracks/algebricks/compiler/rewriter/rulecontrollers/SequentialFixpointRuleController.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-compiler/src/main/java/org/apache/hyracks/algebricks/compiler/rewriter/rulecontrollers/SequentialFixpointRuleController.java b/algebricks/algebricks-compiler/src/main/java/org/apache/hyracks/algebricks/compiler/rewriter/rulecontrollers/SequentialFixpointRuleController.java index 440af4b..b6d6d61 100644 --- a/algebricks/algebricks-compiler/src/main/java/org/apache/hyracks/algebricks/compiler/rewriter/rulecontrollers/SequentialFixpointRuleController.java +++ b/algebricks/algebricks-compiler/src/main/java/org/apache/hyracks/algebricks/compiler/rewriter/rulecontrollers/SequentialFixpointRuleController.java @@ -59,5 +59,4 @@ public class SequentialFixpointRuleController extends AbstractRuleController { } while (anyChange); return anyRuleFired; } - } http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/base/LogicalOperatorTag.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/base/LogicalOperatorTag.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/base/LogicalOperatorTag.java index 6668536..5388092 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/base/LogicalOperatorTag.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/base/LogicalOperatorTag.java @@ -48,6 +48,7 @@ public enum LogicalOperatorTag { TOKENIZE, UNIONALL, UNNEST, + OUTER_UNNEST, UNNEST_MAP, UPDATE, WRITE, http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/expressions/BroadcastExpressionAnnotation.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/expressions/BroadcastExpressionAnnotation.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/expressions/BroadcastExpressionAnnotation.java index 07124bf..6c47b82 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/expressions/BroadcastExpressionAnnotation.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/expressions/BroadcastExpressionAnnotation.java @@ -46,4 +46,9 @@ public class BroadcastExpressionAnnotation implements IExpressionAnnotation { return bcast; } + @Override + public String toString() { + return BROADCAST_ANNOTATION_KEY; + } + } http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/expressions/IndexedNLJoinExpressionAnnotation.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/expressions/IndexedNLJoinExpressionAnnotation.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/expressions/IndexedNLJoinExpressionAnnotation.java index eaeeac7..a38a96c 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/expressions/IndexedNLJoinExpressionAnnotation.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/expressions/IndexedNLJoinExpressionAnnotation.java @@ -41,4 +41,9 @@ public class IndexedNLJoinExpressionAnnotation implements IExpressionAnnotation clone.setObject(object); return clone; } + + @Override + public String toString() { + return INDEXED_NL_JOIN_ANNOTATION_KEY; + } } http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/AbstractLogicalOperator.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/AbstractLogicalOperator.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/AbstractLogicalOperator.java index 79452c8..cfb64d4 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/AbstractLogicalOperator.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/AbstractLogicalOperator.java @@ -24,7 +24,6 @@ import java.util.List; import java.util.Map; import org.apache.commons.lang3.mutable.Mutable; - import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException; import org.apache.hyracks.algebricks.core.algebra.base.IHyracksJobBuilder; import org.apache.hyracks.algebricks.core.algebra.base.ILogicalOperator; @@ -76,6 +75,7 @@ public abstract class AbstractLogicalOperator implements ILogicalOperator { @Override public abstract LogicalOperatorTag getOperatorTag(); + @Override public ExecutionMode getExecutionMode() { return mode; } @@ -154,7 +154,7 @@ public abstract class AbstractLogicalOperator implements ILogicalOperator { @Override public final void contributeRuntimeOperator(IHyracksJobBuilder builder, JobGenContext context, IOperatorSchema propagatedSchema, IOperatorSchema[] inputSchemas, IOperatorSchema outerPlanSchema) - throws AlgebricksException { + throws AlgebricksException { if (bJobGenEnabled) { if (physicalOperator == null) { throw new AlgebricksException("Physical operator not set for operator: " + this); http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/AbstractUnnestNonMapOperator.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/AbstractUnnestNonMapOperator.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/AbstractUnnestNonMapOperator.java new file mode 100644 index 0000000..c04957c --- /dev/null +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/AbstractUnnestNonMapOperator.java @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.hyracks.algebricks.core.algebra.operators.logical; + +import java.util.ArrayList; + +import org.apache.commons.lang3.mutable.Mutable; +import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException; +import org.apache.hyracks.algebricks.core.algebra.base.ILogicalExpression; +import org.apache.hyracks.algebricks.core.algebra.base.LogicalVariable; +import org.apache.hyracks.algebricks.core.algebra.properties.VariablePropagationPolicy; +import org.apache.hyracks.algebricks.runtime.base.IUnnestingPositionWriter; + +public abstract class AbstractUnnestNonMapOperator extends AbstractUnnestOperator { + + protected LogicalVariable positionalVariable; + + /** + * Used to set the position offset for positional variable + */ + protected ILogicalExpression positionOffsetExpr; + + /** + * Specify the writer of the positional variable + */ + protected IUnnestingPositionWriter positionWriter; + + /** + * Specify the type of the positional variable + */ + protected Object positionalVariableType; + + public AbstractUnnestNonMapOperator(LogicalVariable variable, Mutable expression) { + super(makeSingletonList(variable), expression); + } + + public AbstractUnnestNonMapOperator(LogicalVariable variable, Mutable expression, + LogicalVariable positionalVariable, Object positionalVariableType, + IUnnestingPositionWriter positionWriter) { + this(variable, expression); + this.setPositionalVariable(positionalVariable); + this.setPositionalVariableType(positionalVariableType); + this.setPositionWriter(positionWriter); + } + + public LogicalVariable getVariable() { + return variables.get(0); + } + + public void setPositionalVariable(LogicalVariable positionalVariable) { + this.positionalVariable = positionalVariable; + } + + public LogicalVariable getPositionalVariable() { + return positionalVariable; + } + + public void setPositionWriter(IUnnestingPositionWriter positionWriter) { + this.positionWriter = positionWriter; + } + + public IUnnestingPositionWriter getPositionWriter() { + return positionalVariable != null ? positionWriter : null; + } + + public void setPositionalVariableType(Object positionalVariableType) { + this.positionalVariableType = positionalVariableType; + } + + public Object getPositionalVariableType() { + return positionalVariableType; + } + + public void setPositionOffsetExpr(ILogicalExpression posOffsetExpr) { + this.positionOffsetExpr = posOffsetExpr; + } + + public ILogicalExpression getPositionOffsetExpr() { + return this.positionOffsetExpr; + } + + protected static ArrayList makeSingletonList(E item) { + ArrayList array = new ArrayList(1); + array.add(item); + return array; + } + + @Override + public VariablePropagationPolicy getVariablePropagationPolicy() { + return new VariablePropagationPolicy() { + + @Override + public void propagateVariables(IOperatorSchema target, IOperatorSchema... sources) + throws AlgebricksException { + if (sources.length > 0) { + target.addAllVariables(sources[0]); + } + for (LogicalVariable v : variables) { + target.addVariable(v); + } + if (positionalVariable != null) { + target.addVariable(positionalVariable); + } + } + }; + } + +} http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/OuterUnnestOperator.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/OuterUnnestOperator.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/OuterUnnestOperator.java new file mode 100644 index 0000000..133656b --- /dev/null +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/OuterUnnestOperator.java @@ -0,0 +1,71 @@ +/* + * Copyright 2009-2013 by The Regents of the University of California + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * you may obtain a copy of the License from + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hyracks.algebricks.core.algebra.operators.logical; + +import java.util.ArrayList; +import java.util.List; + +import org.apache.commons.lang3.mutable.Mutable; +import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException; +import org.apache.hyracks.algebricks.core.algebra.base.ILogicalExpression; +import org.apache.hyracks.algebricks.core.algebra.base.LogicalOperatorTag; +import org.apache.hyracks.algebricks.core.algebra.base.LogicalVariable; +import org.apache.hyracks.algebricks.core.algebra.expressions.IVariableTypeEnvironment; +import org.apache.hyracks.algebricks.core.algebra.typing.ITypingContext; +import org.apache.hyracks.algebricks.core.algebra.typing.PropagatingTypeEnvironment; +import org.apache.hyracks.algebricks.core.algebra.visitors.ILogicalOperatorVisitor; +import org.apache.hyracks.algebricks.runtime.base.IUnnestingPositionWriter; + +public class OuterUnnestOperator extends AbstractUnnestNonMapOperator { + + public OuterUnnestOperator(LogicalVariable variable, Mutable expression) { + super(variable, expression); + } + + public OuterUnnestOperator(LogicalVariable variable, Mutable expression, + LogicalVariable positionalVariable, Object positionalVariableType, + IUnnestingPositionWriter positionWriter) { + super(variable, expression, positionalVariable, positionalVariableType, positionWriter); + } + + @Override + public R accept(ILogicalOperatorVisitor visitor, T arg) throws AlgebricksException { + return visitor.visitOuterUnnestOperator(this, arg); + } + + @Override + public IVariableTypeEnvironment computeOutputTypeEnvironment(ITypingContext ctx) throws AlgebricksException { + PropagatingTypeEnvironment env = createPropagatingAllInputsTypeEnvironment(ctx); + Object t = env.getType(expression.getValue()); + env.setVarType(variables.get(0), t); + if (positionalVariable != null) { + env.setVarType(positionalVariable, positionalVariableType); + } + + // The produced variables of the this operator are nullable because of the left outer semantics. + List nullableVars = new ArrayList(); + nullableVars.add(variables.get(0)); + if (positionalVariable != null) { + nullableVars.add(positionalVariable); + } + env.getCorrelatedNullableVariableLists().add(nullableVars); + return env; + } + + @Override + public LogicalOperatorTag getOperatorTag() { + return LogicalOperatorTag.OUTER_UNNEST; + } +} http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/SinkOperator.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/SinkOperator.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/SinkOperator.java index bfd2df7..af85fdd 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/SinkOperator.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/SinkOperator.java @@ -19,10 +19,8 @@ package org.apache.hyracks.algebricks.core.algebra.operators.logical; import java.util.ArrayList; -import java.util.List; import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException; -import org.apache.hyracks.algebricks.common.utils.Triple; import org.apache.hyracks.algebricks.core.algebra.base.LogicalOperatorTag; import org.apache.hyracks.algebricks.core.algebra.base.LogicalVariable; import org.apache.hyracks.algebricks.core.algebra.expressions.IVariableTypeEnvironment; @@ -30,7 +28,6 @@ import org.apache.hyracks.algebricks.core.algebra.properties.TypePropagationPoli import org.apache.hyracks.algebricks.core.algebra.properties.VariablePropagationPolicy; import org.apache.hyracks.algebricks.core.algebra.typing.ITypeEnvPointer; import org.apache.hyracks.algebricks.core.algebra.typing.ITypingContext; -import org.apache.hyracks.algebricks.core.algebra.typing.NonPropagatingTypeEnvironment; import org.apache.hyracks.algebricks.core.algebra.typing.OpRefTypeEnvPointer; import org.apache.hyracks.algebricks.core.algebra.typing.PropagatingTypeEnvironment; import org.apache.hyracks.algebricks.core.algebra.visitors.ILogicalExpressionReferenceTransform; @@ -44,14 +41,15 @@ public class SinkOperator extends AbstractLogicalOperator { for (int i = 0; i < inputs.size(); i++) { for (LogicalVariable v : inputs.get(i).getValue().getSchema()) { if (!schema.contains(v)) - schema.add(v); + schema.add(v); } } } @Override - public boolean acceptExpressionTransform(ILogicalExpressionReferenceTransform transform) throws AlgebricksException { + public boolean acceptExpressionTransform(ILogicalExpressionReferenceTransform transform) + throws AlgebricksException { return false; } @@ -72,13 +70,13 @@ public class SinkOperator extends AbstractLogicalOperator { @Override public IVariableTypeEnvironment computeOutputTypeEnvironment(ITypingContext ctx) throws AlgebricksException { - ITypeEnvPointer[] envPointers = new ITypeEnvPointer[inputs.size()]; - for (int i = 0; i < inputs.size(); i++) { - envPointers[i] = new OpRefTypeEnvPointer(inputs.get(i), ctx); - } - PropagatingTypeEnvironment env = new PropagatingTypeEnvironment(ctx.getExpressionTypeComputer(), - ctx.getNullableTypeComputer(), ctx.getMetadataProvider(), TypePropagationPolicy.ALL, envPointers); - return env; + ITypeEnvPointer[] envPointers = new ITypeEnvPointer[inputs.size()]; + for (int i = 0; i < inputs.size(); i++) { + envPointers[i] = new OpRefTypeEnvPointer(inputs.get(i), ctx); + } + PropagatingTypeEnvironment env = new PropagatingTypeEnvironment(ctx.getExpressionTypeComputer(), + ctx.getNullableTypeComputer(), ctx.getMetadataProvider(), TypePropagationPolicy.ALL, envPointers); + return env; } http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/UnnestOperator.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/UnnestOperator.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/UnnestOperator.java index aded8a9..1ee0942 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/UnnestOperator.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/UnnestOperator.java @@ -18,8 +18,6 @@ */ package org.apache.hyracks.algebricks.core.algebra.operators.logical; -import java.util.ArrayList; - import org.apache.commons.lang3.mutable.Mutable; import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException; @@ -27,81 +25,19 @@ import org.apache.hyracks.algebricks.core.algebra.base.ILogicalExpression; import org.apache.hyracks.algebricks.core.algebra.base.LogicalOperatorTag; import org.apache.hyracks.algebricks.core.algebra.base.LogicalVariable; import org.apache.hyracks.algebricks.core.algebra.expressions.IVariableTypeEnvironment; -import org.apache.hyracks.algebricks.core.algebra.properties.VariablePropagationPolicy; import org.apache.hyracks.algebricks.core.algebra.typing.ITypingContext; import org.apache.hyracks.algebricks.core.algebra.visitors.ILogicalOperatorVisitor; import org.apache.hyracks.algebricks.runtime.base.IUnnestingPositionWriter; -public class UnnestOperator extends AbstractUnnestOperator { - - private LogicalVariable positionalVariable; - - /** - * Used to set the position offset for positional variable - */ - private ILogicalExpression positionOffsetExpr; - - /** - * Specify the writer of the positional variable - */ - private IUnnestingPositionWriter positionWriter; - - /** - * Specify the type of the positional variable - */ - private Object positionalVariableType; +public class UnnestOperator extends AbstractUnnestNonMapOperator { public UnnestOperator(LogicalVariable variable, Mutable expression) { - super(makeSingletonList(variable), expression); + super(variable, expression); } public UnnestOperator(LogicalVariable variable, Mutable expression, LogicalVariable positionalVariable, Object positionalVariableType, IUnnestingPositionWriter positionWriter) { - this(variable, expression); - this.setPositionalVariable(positionalVariable); - this.setPositionalVariableType(positionalVariableType); - this.setPositionWriter(positionWriter); - } - - @Override - public LogicalOperatorTag getOperatorTag() { - return LogicalOperatorTag.UNNEST; - } - - public LogicalVariable getVariable() { - return variables.get(0); - } - - public void setPositionalVariable(LogicalVariable positionalVariable) { - this.positionalVariable = positionalVariable; - } - - public LogicalVariable getPositionalVariable() { - return positionalVariable; - } - - public void setPositionWriter(IUnnestingPositionWriter positionWriter) { - this.positionWriter = positionWriter; - } - - public IUnnestingPositionWriter getPositionWriter() { - return positionalVariable != null ? positionWriter : null; - } - - public void setPositionalVariableType(Object positionalVariableType) { - this.positionalVariableType = positionalVariableType; - } - - public Object getPositionalVariableType() { - return positionalVariableType; - } - - public void setPositionOffsetExpr(ILogicalExpression posOffsetExpr) { - this.positionOffsetExpr = posOffsetExpr; - } - - public ILogicalExpression getPositionOffsetExpr() { - return this.positionOffsetExpr; + super(variable, expression, positionalVariable, positionalVariableType, positionWriter); } @Override @@ -109,12 +45,6 @@ public class UnnestOperator extends AbstractUnnestOperator { return visitor.visitUnnestOperator(this, arg); } - private static ArrayList makeSingletonList(E item) { - ArrayList array = new ArrayList(1); - array.add(item); - return array; - } - @Override public IVariableTypeEnvironment computeOutputTypeEnvironment(ITypingContext ctx) throws AlgebricksException { IVariableTypeEnvironment env = createPropagatingAllInputsTypeEnvironment(ctx); @@ -127,22 +57,7 @@ public class UnnestOperator extends AbstractUnnestOperator { } @Override - public VariablePropagationPolicy getVariablePropagationPolicy() { - return new VariablePropagationPolicy() { - - @Override - public void propagateVariables(IOperatorSchema target, IOperatorSchema... sources) - throws AlgebricksException { - if (sources.length > 0) { - target.addAllVariables(sources[0]); - } - for (LogicalVariable v : variables) { - target.addVariable(v); - } - if (positionalVariable != null) { - target.addVariable(positionalVariable); - } - } - }; + public LogicalOperatorTag getOperatorTag() { + return LogicalOperatorTag.UNNEST; } } \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/FDsAndEquivClassesVisitor.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/FDsAndEquivClassesVisitor.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/FDsAndEquivClassesVisitor.java index 6d00fab..25993bb 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/FDsAndEquivClassesVisitor.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/FDsAndEquivClassesVisitor.java @@ -29,7 +29,6 @@ import java.util.Map.Entry; import java.util.Set; import org.apache.commons.lang3.mutable.Mutable; - import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException; import org.apache.hyracks.algebricks.common.exceptions.NotImplementedException; import org.apache.hyracks.algebricks.common.utils.Pair; @@ -65,6 +64,7 @@ import org.apache.hyracks.algebricks.core.algebra.operators.logical.LimitOperato import org.apache.hyracks.algebricks.core.algebra.operators.logical.MaterializeOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.NestedTupleSourceOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.OrderOperator; +import org.apache.hyracks.algebricks.core.algebra.operators.logical.OuterUnnestOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.PartitioningSplitOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.ProjectOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.ReplicateOperator; @@ -311,9 +311,9 @@ public class FDsAndEquivClassesVisitor implements ILogicalOperatorVisitor>>> Group-by list changed from " - + GroupByOperator.veListToString(gByList) + " to " + GroupByOperator.veListToString(newGbyList) - + ".\n"); + AlgebricksConfig.ALGEBRICKS_LOGGER + .fine(">>>> Group-by list changed from " + GroupByOperator.veListToString(gByList) + " to " + + GroupByOperator.veListToString(newGbyList) + ".\n"); } gByList.clear(); gByList.addAll(newGbyList); @@ -509,7 +509,8 @@ public class FDsAndEquivClassesVisitor implements ILogicalOperatorVisitor newVarEqcMap = new HashMap(); for (Entry entry : eqClasses.entrySet()) { EquivalenceClass eqc = entry.getValue(); // If the equivalence class contains the right-hand-side expression, // the left-hand-side variable is added into the equivalence class. if (eqc.contains(expr)) { eqc.addMember(var); + newVarEqcMap.put(var, eqc); // Add var as a map key for the equivalence class. } } + eqClasses.putAll(newVarEqcMap); } } -} \ No newline at end of file + @Override + public Void visitOuterUnnestOperator(OuterUnnestOperator op, IOptimizationContext ctx) throws AlgebricksException { + propagateFDsAndEquivClasses(op, ctx); + return null; + } + +} http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java index a982ed4..dc535ea 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java @@ -54,6 +54,7 @@ import org.apache.hyracks.algebricks.core.algebra.operators.logical.MaterializeO import org.apache.hyracks.algebricks.core.algebra.operators.logical.NestedTupleSourceOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.OrderOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.OrderOperator.IOrder; +import org.apache.hyracks.algebricks.core.algebra.operators.logical.OuterUnnestOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.PartitioningSplitOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.ProjectOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.ReplicateOperator; @@ -576,4 +577,18 @@ public class IsomorphismOperatorVisitor implements ILogicalOperatorVisitor newInputList = new ArrayList(); newInputList.addAll(op.getVariables()); - return new UnnestMapOperator(newInputList, deepCopyExpressionRef(op.getExpressionRef()), new ArrayList( - op.getVariableTypes()), op.propagatesInput()); + return new UnnestMapOperator(newInputList, deepCopyExpressionRef(op.getExpressionRef()), + new ArrayList(op.getVariableTypes()), op.propagatesInput()); } @Override @@ -272,7 +272,8 @@ public class OperatorDeepCopyVisitor implements ILogicalOperatorVisitor> newLSMComponentFilterExpressions = new ArrayList>(); deepCopyExpressionRefs(newKeyExpressions, op.getAdditionalFilteringExpressions()); InsertDeleteOperator insertDeleteOp = new InsertDeleteOperator(op.getDataSource(), - deepCopyExpressionRef(op.getPayloadExpression()), newKeyExpressions, op.getOperation(), op.isBulkload()); + deepCopyExpressionRef(op.getPayloadExpression()), newKeyExpressions, op.getOperation(), + op.isBulkload()); insertDeleteOp.setAdditionalFilteringExpressions(newLSMComponentFilterExpressions); return insertDeleteOp; } @@ -322,12 +323,13 @@ public class OperatorDeepCopyVisitor implements ILogicalOperatorVisitor> newExprs, List> oldExprs) { for (Mutable oldExpr : oldExprs) - newExprs.add(new MutableObject(((AbstractLogicalExpression) oldExpr.getValue()) - .cloneExpression())); + newExprs.add(new MutableObject( + ((AbstractLogicalExpression) oldExpr.getValue()).cloneExpression())); } private Mutable deepCopyExpressionRef(Mutable oldExpr) { - return new MutableObject(((AbstractLogicalExpression) oldExpr.getValue()).cloneExpression()); + return new MutableObject( + ((AbstractLogicalExpression) oldExpr.getValue()).cloneExpression()); } private List deepCopyVars(List newVars, List oldVars) { @@ -346,8 +348,8 @@ public class OperatorDeepCopyVisitor implements ILogicalOperatorVisitor>> ordersAndExprs) { List>> newOrdersAndExprs = new ArrayList>>(); for (Pair> pair : ordersAndExprs) - newOrdersAndExprs.add(new Pair>(pair.first, - deepCopyExpressionRef(pair.second))); + newOrdersAndExprs + .add(new Pair>(pair.first, deepCopyExpressionRef(pair.second))); return newOrdersAndExprs; } @@ -369,4 +371,10 @@ public class OperatorDeepCopyVisitor implements ILogicalOperatorVisitor pair) throws AlgebricksException { - List variables = op.getVariables(); - for (int i = 0; i < variables.size(); i++) { - if (variables.get(i) == pair.first) { - variables.set(i, pair.second); - return null; - } - } - op.getExpressionRef().getValue().substituteVar(pair.first, pair.second); - substVarTypes(op, pair); - return null; + return visitUnnestNonMapOperator(op, pair); } @Override @@ -470,4 +463,24 @@ public class SubstituteVariableVisitor implements ILogicalOperatorVisitor pair) + throws AlgebricksException { + return visitUnnestNonMapOperator(op, pair); + } + + private Void visitUnnestNonMapOperator(AbstractUnnestNonMapOperator op, Pair pair) + throws AlgebricksException { + List variables = op.getVariables(); + for (int i = 0; i < variables.size(); i++) { + if (variables.get(i) == pair.first) { + variables.set(i, pair.second); + return null; + } + } + op.getExpressionRef().getValue().substituteVar(pair.first, pair.second); + substVarTypes(op, pair); + return null; + } } http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java index bfb9036..f80e1cd 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java @@ -50,6 +50,7 @@ import org.apache.hyracks.algebricks.core.algebra.operators.logical.MaterializeO import org.apache.hyracks.algebricks.core.algebra.operators.logical.NestedTupleSourceOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.OrderOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.OrderOperator.IOrder; +import org.apache.hyracks.algebricks.core.algebra.operators.logical.OuterUnnestOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.PartitioningSplitOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.ProjectOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.ReplicateOperator; @@ -408,4 +409,10 @@ public class UsedVariableVisitor implements ILogicalOperatorVisitor return null; } + @Override + public Void visitOuterUnnestOperator(OuterUnnestOperator op, Void arg) throws AlgebricksException { + op.getExpressionRef().getValue().getUsedVariables(usedVariables); + return null; + } + } http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/prettyprint/LogicalOperatorPrettyPrintVisitor.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/prettyprint/LogicalOperatorPrettyPrintVisitor.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/prettyprint/LogicalOperatorPrettyPrintVisitor.java index 1404a47..17f2285 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/prettyprint/LogicalOperatorPrettyPrintVisitor.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/prettyprint/LogicalOperatorPrettyPrintVisitor.java @@ -21,7 +21,6 @@ package org.apache.hyracks.algebricks.core.algebra.prettyprint; import java.util.List; import org.apache.commons.lang3.mutable.Mutable; - import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException; import org.apache.hyracks.algebricks.common.utils.Pair; import org.apache.hyracks.algebricks.common.utils.Triple; @@ -48,6 +47,7 @@ import org.apache.hyracks.algebricks.core.algebra.operators.logical.LimitOperato import org.apache.hyracks.algebricks.core.algebra.operators.logical.MaterializeOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.NestedTupleSourceOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.OrderOperator; +import org.apache.hyracks.algebricks.core.algebra.operators.logical.OuterUnnestOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.PartitioningSplitOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.ProjectOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.ReplicateOperator; @@ -86,7 +86,8 @@ public class LogicalOperatorPrettyPrintVisitor implements ILogicalOperatorVisito } @Override - public String visitRunningAggregateOperator(RunningAggregateOperator op, Integer indent) throws AlgebricksException { + public String visitRunningAggregateOperator(RunningAggregateOperator op, Integer indent) + throws AlgebricksException { StringBuilder buffer = new StringBuilder(); addIndent(buffer, indent).append("running-aggregate ").append(op.getVariables()).append(" <- "); pprintExprList(op.getExpressions(), buffer, indent); @@ -185,7 +186,8 @@ public class LogicalOperatorPrettyPrintVisitor implements ILogicalOperatorVisito } @Override - public String visitDistributeResultOperator(DistributeResultOperator op, Integer indent) throws AlgebricksException { + public String visitDistributeResultOperator(DistributeResultOperator op, Integer indent) + throws AlgebricksException { StringBuilder buffer = new StringBuilder(); addIndent(buffer, indent).append("distribute result "); pprintExprList(op.getExpressions(), buffer, indent); @@ -256,11 +258,21 @@ public class LogicalOperatorPrettyPrintVisitor implements ILogicalOperatorVisito } @Override + public String visitOuterUnnestOperator(OuterUnnestOperator op, Integer indent) throws AlgebricksException { + StringBuilder buffer = new StringBuilder(); + addIndent(buffer, indent).append("outer-unnest " + op.getVariable()); + if (op.getPositionalVariable() != null) { + buffer.append(" at " + op.getPositionalVariable()); + } + buffer.append(" <- " + op.getExpressionRef().getValue().accept(exprVisitor, indent)); + return buffer.toString(); + } + + @Override public String visitUnnestMapOperator(UnnestMapOperator op, Integer indent) throws AlgebricksException { StringBuilder buffer = new StringBuilder(); - addIndent(buffer, indent).append( - "unnest-map " + op.getVariables() + " <- " - + op.getExpressionRef().getValue().accept(exprVisitor, indent)); + addIndent(buffer, indent).append("unnest-map " + op.getVariables() + " <- " + + op.getExpressionRef().getValue().accept(exprVisitor, indent)); return buffer.toString(); } @@ -293,8 +305,8 @@ public class LogicalOperatorPrettyPrintVisitor implements ILogicalOperatorVisito @Override public String visitScriptOperator(ScriptOperator op, Integer indent) { StringBuilder buffer = new StringBuilder(); - addIndent(buffer, indent).append( - "script (in: " + op.getInputVariables() + ") (out: " + op.getOutputVariables() + ")"); + addIndent(buffer, indent) + .append("script (in: " + op.getInputVariables() + ") (out: " + op.getOutputVariables() + ")"); return buffer.toString(); } @@ -427,8 +439,8 @@ public class LogicalOperatorPrettyPrintVisitor implements ILogicalOperatorVisito public String visitExternalDataLookupOperator(ExternalDataLookupOperator op, Integer indent) throws AlgebricksException { StringBuilder buffer = new StringBuilder(); - addIndent(buffer, indent).append( - "external-instant-lookup " + op.getVariables() + " <- " + op.getExpressionRef().getValue()); + addIndent(buffer, indent) + .append("external-instant-lookup " + op.getVariables() + " <- " + op.getExpressionRef().getValue()); return buffer.toString(); } http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/properties/TypePropagationPolicy.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/properties/TypePropagationPolicy.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/properties/TypePropagationPolicy.java index 612dce2..fa9ad05 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/properties/TypePropagationPolicy.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/properties/TypePropagationPolicy.java @@ -36,7 +36,8 @@ public abstract class TypePropagationPolicy { for (ITypeEnvPointer p : typeEnvs) { IVariableTypeEnvironment env = p.getTypeEnv(); if (env == null) { - throw new AlgebricksException("Null environment for pointer " + p + " in getVarType for var=" + var); + throw new AlgebricksException( + "Null environment for pointer " + p + " in getVarType for var=" + var); } Object t = env.getVarType(var, nonNullVariableList, correlatedNullableVariableLists); if (t != null) { http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/visitors/ILogicalOperatorVisitor.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/visitors/ILogicalOperatorVisitor.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/visitors/ILogicalOperatorVisitor.java index 423a0a4..285812c 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/visitors/ILogicalOperatorVisitor.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/visitors/ILogicalOperatorVisitor.java @@ -37,6 +37,7 @@ import org.apache.hyracks.algebricks.core.algebra.operators.logical.LimitOperato import org.apache.hyracks.algebricks.core.algebra.operators.logical.MaterializeOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.NestedTupleSourceOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.OrderOperator; +import org.apache.hyracks.algebricks.core.algebra.operators.logical.OuterUnnestOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.PartitioningSplitOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.ProjectOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.ReplicateOperator; @@ -96,6 +97,8 @@ public interface ILogicalOperatorVisitor { public R visitUnnestOperator(UnnestOperator op, T arg) throws AlgebricksException; + public R visitOuterUnnestOperator(OuterUnnestOperator op, T arg) throws AlgebricksException; + public R visitUnnestMapOperator(UnnestMapOperator op, T arg) throws AlgebricksException; public R visitDataScanOperator(DataSourceScanOperator op, T arg) throws AlgebricksException; http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/rewriter/base/AbstractRuleController.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/rewriter/base/AbstractRuleController.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/rewriter/base/AbstractRuleController.java index e3695fd..5adb566 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/rewriter/base/AbstractRuleController.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/rewriter/base/AbstractRuleController.java @@ -22,7 +22,6 @@ import java.util.Collection; import java.util.logging.Level; import org.apache.commons.lang3.mutable.Mutable; - import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException; import org.apache.hyracks.algebricks.core.algebra.base.ILogicalOperator; import org.apache.hyracks.algebricks.core.algebra.base.ILogicalPlan; @@ -46,7 +45,7 @@ public abstract class AbstractRuleController { /** * Each rewriting strategy may differ in the - * + * * @param root * @param ruleClasses * @return true iff one of the rules in the collection fired http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/rewriter/base/IAlgebraicRewriteRule.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/rewriter/base/IAlgebraicRewriteRule.java b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/rewriter/base/IAlgebraicRewriteRule.java index ad68348..83740de 100644 --- a/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/rewriter/base/IAlgebraicRewriteRule.java +++ b/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/rewriter/base/IAlgebraicRewriteRule.java @@ -19,7 +19,6 @@ package org.apache.hyracks.algebricks.core.rewriter.base; import org.apache.commons.lang3.mutable.Mutable; - import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException; import org.apache.hyracks.algebricks.core.algebra.base.ILogicalOperator; import org.apache.hyracks.algebricks.core.algebra.base.IOptimizationContext; http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/IntroHashPartitionMergeExchange.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/IntroHashPartitionMergeExchange.java b/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/IntroHashPartitionMergeExchange.java deleted file mode 100644 index 959f887..0000000 --- a/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/IntroHashPartitionMergeExchange.java +++ /dev/null @@ -1,78 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.hyracks.algebricks.rewriter.rules; - -import java.util.ArrayList; -import java.util.List; - -import org.apache.commons.lang3.mutable.Mutable; - -import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException; -import org.apache.hyracks.algebricks.core.algebra.base.ILogicalOperator; -import org.apache.hyracks.algebricks.core.algebra.base.IOptimizationContext; -import org.apache.hyracks.algebricks.core.algebra.base.PhysicalOperatorTag; -import org.apache.hyracks.algebricks.core.algebra.operators.logical.AbstractLogicalOperator; -import org.apache.hyracks.algebricks.core.algebra.operators.physical.HashPartitionExchangePOperator; -import org.apache.hyracks.algebricks.core.algebra.operators.physical.HashPartitionMergeExchangePOperator; -import org.apache.hyracks.algebricks.core.algebra.operators.physical.SortMergeExchangePOperator; -import org.apache.hyracks.algebricks.core.algebra.properties.OrderColumn; -import org.apache.hyracks.algebricks.core.rewriter.base.IAlgebraicRewriteRule; - -public class IntroHashPartitionMergeExchange implements IAlgebraicRewriteRule { - - @Override - public boolean rewritePre(Mutable opRef, IOptimizationContext context) throws AlgebricksException { - return false; - } - - @Override - public boolean rewritePost(Mutable opRef, IOptimizationContext context) - throws AlgebricksException { - AbstractLogicalOperator op1 = (AbstractLogicalOperator) opRef.getValue(); - if (op1.getPhysicalOperator() == null - || (op1.getPhysicalOperator().getOperatorTag() != PhysicalOperatorTag.HASH_PARTITION_EXCHANGE && op1 - .getPhysicalOperator().getOperatorTag() != PhysicalOperatorTag.HASH_PARTITION_MERGE_EXCHANGE)) { - return false; - } - AbstractLogicalOperator op2 = (AbstractLogicalOperator) op1.getInputs().get(0).getValue(); - if (op2.getPhysicalOperator() == null - || op2.getPhysicalOperator().getOperatorTag() != PhysicalOperatorTag.SORT_MERGE_EXCHANGE) { - return false; - } - if (op1.getPhysicalOperator().getOperatorTag() == PhysicalOperatorTag.HASH_PARTITION_MERGE_EXCHANGE) { - // if it is a hash_partition_merge_exchange, the sort_merge_exchange can be simply removed - op1.getInputs().get(0).setValue(op2.getInputs().get(0).getValue()); - op1.computeDeliveredPhysicalProperties(context); - return true; - } - HashPartitionExchangePOperator hpe = (HashPartitionExchangePOperator) op1.getPhysicalOperator(); - SortMergeExchangePOperator sme = (SortMergeExchangePOperator) op2.getPhysicalOperator(); - List ocList = new ArrayList(); - for (OrderColumn oc : sme.getSortColumns()) { - ocList.add(oc); - } - HashPartitionMergeExchangePOperator hpme = new HashPartitionMergeExchangePOperator(ocList, hpe.getHashFields(), - hpe.getDomain()); - op1.setPhysicalOperator(hpme); - op1.getInputs().get(0).setValue(op2.getInputs().get(0).getValue()); - op1.computeDeliveredPhysicalProperties(context); - return true; - } - -} http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/NestedSubplanToJoinRule.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/NestedSubplanToJoinRule.java b/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/NestedSubplanToJoinRule.java index 113a06e..f8a93b0 100644 --- a/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/NestedSubplanToJoinRule.java +++ b/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/NestedSubplanToJoinRule.java @@ -25,7 +25,6 @@ import java.util.Set; import org.apache.commons.lang3.mutable.Mutable; import org.apache.commons.lang3.mutable.MutableObject; - import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException; import org.apache.hyracks.algebricks.core.algebra.base.ILogicalExpression; import org.apache.hyracks.algebricks.core.algebra.base.ILogicalOperator; @@ -37,20 +36,21 @@ import org.apache.hyracks.algebricks.core.algebra.expressions.ConstantExpression import org.apache.hyracks.algebricks.core.algebra.operators.logical.AbstractLogicalOperator; import org.apache.hyracks.algebricks.core.algebra.operators.logical.AbstractOperatorWithNestedPlans; import org.apache.hyracks.algebricks.core.algebra.operators.logical.EmptyTupleSourceOperator; -import org.apache.hyracks.algebricks.core.algebra.operators.logical.LeftOuterJoinOperator; +import org.apache.hyracks.algebricks.core.algebra.operators.logical.InnerJoinOperator; import org.apache.hyracks.algebricks.core.algebra.util.OperatorPropertiesUtil; import org.apache.hyracks.algebricks.core.rewriter.base.IAlgebraicRewriteRule; /** * replace Subplan operators with nested loop joins where the join condition is true, if the Subplan * does not contain free variables (does not have correlations to the input stream). - * + * * @author yingyib */ public class NestedSubplanToJoinRule implements IAlgebraicRewriteRule { @Override - public boolean rewritePre(Mutable opRef, IOptimizationContext context) throws AlgebricksException { + public boolean rewritePre(Mutable opRef, IOptimizationContext context) + throws AlgebricksException { return false; } @@ -103,17 +103,20 @@ public class NestedSubplanToJoinRule implements IAlgebraicRewriteRule { continue; } - /** expend the input and roots into a DAG of nested loop joins */ + /** + * Expends the input and roots into a DAG of nested loop joins. + * Though joins should be left-outer joins, a left-outer join with condition TRUE is equivalent to an inner join. + **/ Mutable expr = new MutableObject(ConstantExpression.TRUE); Mutable nestedRootRef = nestedRoots.get(0); - ILogicalOperator join = new LeftOuterJoinOperator(expr, new MutableObject(subplanInput), + ILogicalOperator join = new InnerJoinOperator(expr, new MutableObject(subplanInput), nestedRootRef); /** rewrite the nested tuple source to be empty tuple source */ - rewriteNestedTupleSource(nestedRootRef); + rewriteNestedTupleSource(nestedRootRef, context); for (int i = 1; i < nestedRoots.size(); i++) { - join = new LeftOuterJoinOperator(expr, new MutableObject(join), nestedRoots.get(i)); + join = new InnerJoinOperator(expr, new MutableObject(join), nestedRoots.get(i)); } op1.getInputs().get(index).setValue(join); context.computeAndSetTypeEnvironmentForOperator(join); @@ -124,17 +127,20 @@ public class NestedSubplanToJoinRule implements IAlgebraicRewriteRule { /** * rewrite NestedTupleSource operators to EmptyTupleSource operators - * + * * @param nestedRootRef */ - private void rewriteNestedTupleSource(Mutable nestedRootRef) { + private void rewriteNestedTupleSource(Mutable nestedRootRef, IOptimizationContext context) + throws AlgebricksException { AbstractLogicalOperator nestedRoot = (AbstractLogicalOperator) nestedRootRef.getValue(); if (nestedRoot.getOperatorTag() == LogicalOperatorTag.NESTEDTUPLESOURCE) { - nestedRootRef.setValue(new EmptyTupleSourceOperator()); + ILogicalOperator ets = new EmptyTupleSourceOperator(); + nestedRootRef.setValue(ets); + context.computeAndSetTypeEnvironmentForOperator(ets); } List> inputs = nestedRoot.getInputs(); for (Mutable input : inputs) { - rewriteNestedTupleSource(input); + rewriteNestedTupleSource(input, context); } } } http://git-wip-us.apache.org/repos/asf/incubator-asterixdb-hyracks/blob/f9f918a1/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/PushSortDownRule.java ---------------------------------------------------------------------- diff --git a/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/PushSortDownRule.java b/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/PushSortDownRule.java new file mode 100644 index 0000000..7a7003b --- /dev/null +++ b/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/PushSortDownRule.java @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.hyracks.algebricks.rewriter.rules; + +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import org.apache.commons.lang3.mutable.Mutable; +import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException; +import org.apache.hyracks.algebricks.common.utils.Pair; +import org.apache.hyracks.algebricks.core.algebra.base.ILogicalExpression; +import org.apache.hyracks.algebricks.core.algebra.base.ILogicalOperator; +import org.apache.hyracks.algebricks.core.algebra.base.IOptimizationContext; +import org.apache.hyracks.algebricks.core.algebra.base.LogicalOperatorTag; +import org.apache.hyracks.algebricks.core.algebra.base.LogicalVariable; +import org.apache.hyracks.algebricks.core.algebra.operators.logical.OrderOperator; +import org.apache.hyracks.algebricks.core.algebra.operators.logical.OrderOperator.IOrder; +import org.apache.hyracks.algebricks.core.algebra.operators.logical.visitors.VariableUtilities; +import org.apache.hyracks.algebricks.core.rewriter.base.IAlgebraicRewriteRule; + +/** + * This rule pushes down the sort operator + * as much as possible to where the sort keys are available. + * The rule pushes the sort operator down one-step-at-a-time. + */ +public class PushSortDownRule implements IAlgebraicRewriteRule { + + @Override + public boolean rewritePre(Mutable opRef, IOptimizationContext context) + throws AlgebricksException { + ILogicalOperator operator = opRef.getValue(); + if (operator.getOperatorTag() != LogicalOperatorTag.ORDER) { + return false; + } + + // Gets used variables in the sort operator. + OrderOperator orderOperator = (OrderOperator) operator; + List>> orderKeys = orderOperator.getOrderExpressions(); + Set orderUsedVars = new HashSet(); + for (Pair> orderKey : orderKeys) { + orderKey.second.getValue().getUsedVariables(orderUsedVars); + } + Mutable inputOpRef = orderOperator.getInputs().get(0); + ILogicalOperator inputOperator = inputOpRef.getValue(); + + // Only pushes sort through assign: + // 1. Blocking operators like sort/group/join cannot be pushed through. + // 2. Data reducing operators like select/project should not be pushed through. + // 3. Order-destroying operator like unnest/unnest-map cannot be pushed through. + if (inputOperator.getOperatorTag() != LogicalOperatorTag.ASSIGN) { + return false; + } + Set inputProducedVars = new HashSet(); + VariableUtilities.getProducedVariables(inputOperator, inputProducedVars); + + // Intersects used variables in the sort and variables produced by inputOperator. + orderUsedVars.retainAll(inputProducedVars); + if (!orderUsedVars.isEmpty()) { + // If the sort uses any variable that is produced by this operator. + return false; + } + + // Switches sort and its input operator. + opRef.setValue(inputOperator); + inputOpRef.setValue(inputOperator.getInputs().get(0).getValue()); + inputOperator.getInputs().get(0).setValue(orderOperator); + + // Re-computes the type environments. + context.computeAndSetTypeEnvironmentForOperator(orderOperator); + context.computeAndSetTypeEnvironmentForOperator(inputOperator); + return true; + } + + @Override + public boolean rewritePost(Mutable opRef, IOptimizationContext context) + throws AlgebricksException { + return false; + } + +}