Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C647A200C0E for ; Wed, 1 Feb 2017 23:21:47 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id C4956160B46; Wed, 1 Feb 2017 22:21:47 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 9DFDA160B41 for ; Wed, 1 Feb 2017 23:21:45 +0100 (CET) Received: (qmail 78767 invoked by uid 500); 1 Feb 2017 22:21:44 -0000 Mailing-List: contact dev-help@atlas.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@atlas.incubator.apache.org Delivered-To: mailing list dev@atlas.incubator.apache.org Received: (qmail 78756 invoked by uid 99); 1 Feb 2017 22:21:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Feb 2017 22:21:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 0A832C0115 for ; Wed, 1 Feb 2017 22:21:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -6.219 X-Spam-Level: X-Spam-Status: No, score=-6.219 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id bNbWs2wImmpq for ; Wed, 1 Feb 2017 22:21:32 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with SMTP id 45AAD5F177 for ; Wed, 1 Feb 2017 22:21:31 +0000 (UTC) Received: (qmail 78505 invoked by uid 99); 1 Feb 2017 22:21:15 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Feb 2017 22:21:15 +0000 Received: from [10.22.16.32] (unknown [192.175.27.10]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id BA9D81A002B; Wed, 1 Feb 2017 22:21:14 +0000 (UTC) User-Agent: Microsoft-MacOutlook/f.1e.0.170107 Date: Wed, 01 Feb 2017 14:21:13 -0800 Subject: Re: [5/5] incubator-atlas git commit: ATLAS-1369 - Optimize gremlin queries generated by DSL translator From: Madhan Neethiraj Sender: Madhan Neethiraj To: "jnhagelberg@apache.org" CC: "dev@atlas.incubator.apache.org" Message-ID: <453903D7-FA2F-483D-A287-67C372AEBBB4@hortonworks.com> Thread-Topic: [5/5] incubator-atlas git commit: ATLAS-1369 - Optimize gremlin queries generated by DSL translator References: <203ff468a4a44b258cd2c8a49e61557e@git.apache.org> In-Reply-To: Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: 7bit archived-at: Wed, 01 Feb 2017 22:21:48 -0000 Jeff, Does this patch require JDK8? Compiling with JDK7 fails with the failures like: [ERROR] /Users/mneethiraj/Apache/git/incubator-atlas/repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java:20: error: package java.util.function does not exist [ERROR] import java.util.function.Function; [ERROR] ^ [ERROR] /Users/mneethiraj/Apache/git/incubator-atlas/repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpressionFinder.java:32: error: cannot find symbol [ERROR] private final Function predicate; [ERROR] ^ Can you please review and update to make it compatible with JDK7? Thanks, Madhan On 2/1/17, 1:05 PM, "jnhagelberg@apache.org" wrote: ATLAS-1369 - Optimize gremlin queries generated by DSL translator Project: http://git-wip-us.apache.org/repos/asf/incubator-atlas/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-atlas/commit/aa74c73d Tree: http://git-wip-us.apache.org/repos/asf/incubator-atlas/tree/aa74c73d Diff: http://git-wip-us.apache.org/repos/asf/incubator-atlas/diff/aa74c73d Branch: refs/heads/master Commit: aa74c73d0e9fbfd9c51de9f6cdcdea6141575e8e Parents: f640da7 Author: Jeff Hagelberg Authored: Wed Feb 1 16:03:34 2017 -0500 Committer: Jeff Hagelberg Committed: Wed Feb 1 16:04:39 2017 -0500 ---------------------------------------------------------------------- .../groovy/AbstractFunctionExpression.java | 57 ++ .../atlas/groovy/AbstractGroovyExpression.java | 9 + .../atlas/groovy/ArithmeticExpression.java | 12 + .../apache/atlas/groovy/BinaryExpression.java | 9 +- .../org/apache/atlas/groovy/CastExpression.java | 14 +- .../apache/atlas/groovy/ClosureExpression.java | 90 ++- .../atlas/groovy/CodeBlockExpression.java | 61 -- .../atlas/groovy/ComparisonExpression.java | 12 + .../groovy/ComparisonOperatorExpression.java | 8 + .../apache/atlas/groovy/FieldExpression.java | 21 +- .../atlas/groovy/FunctionCallExpression.java | 88 ++- .../apache/atlas/groovy/GroovyExpression.java | 42 +- .../atlas/groovy/IdentifierExpression.java | 31 + .../apache/atlas/groovy/LabeledExpression.java | 54 ++ .../org/apache/atlas/groovy/ListExpression.java | 12 + .../apache/atlas/groovy/LiteralExpression.java | 25 +- .../apache/atlas/groovy/LogicalExpression.java | 12 + .../apache/atlas/groovy/RangeExpression.java | 62 +- .../atlas/groovy/StatementListExpression.java | 98 +++ .../atlas/groovy/TernaryOperatorExpression.java | 25 +- .../apache/atlas/groovy/TraversalStepType.java | 121 ++++ .../atlas/groovy/TypeCoersionExpression.java | 19 +- .../groovy/VariableAssignmentExpression.java | 16 +- distro/src/conf/atlas-application.properties | 7 + graphdb/titan0/pom.xml | 8 + intg/pom.xml | 1 - pom.xml | 8 +- release-log.txt | 5 +- .../atlas/discovery/DataSetLineageService.java | 7 +- .../gremlin/Gremlin2ExpressionFactory.java | 139 +++- .../gremlin/Gremlin3ExpressionFactory.java | 184 +++-- .../atlas/gremlin/GremlinExpressionFactory.java | 274 +++++-- .../atlas/gremlin/optimizer/AliasFinder.java | 103 +++ .../gremlin/optimizer/CallHierarchyVisitor.java | 62 ++ .../optimizer/ExpandAndsOptimization.java | 130 ++++ .../optimizer/ExpandOrsOptimization.java | 584 +++++++++++++++ .../gremlin/optimizer/ExpressionFinder.java | 69 ++ .../gremlin/optimizer/FunctionGenerator.java | 326 +++++++++ .../gremlin/optimizer/GremlinOptimization.java | 48 ++ .../optimizer/GremlinQueryOptimizer.java | 262 +++++++ .../gremlin/optimizer/HasForbiddenType.java | 52 ++ .../apache/atlas/gremlin/optimizer/IsOr.java | 48 ++ .../atlas/gremlin/optimizer/IsOrParent.java | 60 ++ .../gremlin/optimizer/OptimizationContext.java | 116 +++ .../atlas/gremlin/optimizer/OrderFinder.java | 68 ++ .../gremlin/optimizer/PathExpressionFinder.java | 61 ++ .../atlas/gremlin/optimizer/RangeFinder.java | 68 ++ .../gremlin/optimizer/SplitPointFinder.java | 161 +++++ .../gremlin/optimizer/UpdatedExpressions.java | 45 ++ .../graph/GraphBackedMetadataRepository.java | 23 +- .../graph/GraphToTypedInstanceMapper.java | 13 +- .../util/AtlasRepositoryConfiguration.java | 19 +- .../org/apache/atlas/query/GremlinQuery.scala | 103 ++- .../GraphBackedDiscoveryServiceTest.java | 3 + .../AbstractGremlinQueryOptimizerTest.java | 705 +++++++++++++++++++ .../graph/Gremlin2QueryOptimizerTest.java | 363 ++++++++++ .../graph/Gremlin3QueryOptimizerTest.java | 364 ++++++++++ .../atlas/repository/graph/TestIntSequence.java | 35 + .../org/apache/atlas/query/GremlinTest.scala | 4 +- 59 files changed, 5157 insertions(+), 269 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java b/common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java new file mode 100644 index 0000000..2e2307c --- /dev/null +++ b/common/src/main/java/org/apache/atlas/groovy/AbstractFunctionExpression.java @@ -0,0 +1,57 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.groovy; + +/** + * Base class for all expression that can have a caller. + */ +public abstract class AbstractFunctionExpression extends AbstractGroovyExpression { + + // null for global functions + private GroovyExpression caller; + private TraversalStepType type = TraversalStepType.NONE; + + public AbstractFunctionExpression(GroovyExpression target) { + this.caller = target; + } + + public AbstractFunctionExpression(TraversalStepType type, GroovyExpression target) { + this.caller = target; + this.type = type; + } + + public GroovyExpression getCaller() { + return caller; + } + + public void setCaller(GroovyExpression expr) { + caller = expr; + } + + + public void setType(TraversalStepType type) { + this.type = type; + } + + @Override + public TraversalStepType getType() { + return type; + } + + +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java b/common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java index 49eaae8..e4a7781 100644 --- a/common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/AbstractGroovyExpression.java @@ -33,4 +33,13 @@ public abstract class AbstractGroovyExpression implements GroovyExpression { return ctx.getQuery(); } + @Override + public TraversalStepType getType() { + return TraversalStepType.NONE; + } + + @Override + public GroovyExpression copy() { + return copy(getChildren()); + } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java b/common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java index 0aec5d0..a6e1689 100644 --- a/common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/ArithmeticExpression.java @@ -17,6 +17,8 @@ */ package org.apache.atlas.groovy; +import java.util.List; + import org.apache.atlas.AtlasException; /** @@ -56,4 +58,14 @@ public class ArithmeticExpression extends BinaryExpression { public ArithmeticExpression(GroovyExpression left, ArithmeticOperator op, GroovyExpression right) { super(left, op.getGroovyValue(), right); } + + private ArithmeticExpression(GroovyExpression left, String op, GroovyExpression right) { + super(left, op, right); + } + + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.size() == 2; + return new ArithmeticExpression(newChildren.get(0), op, newChildren.get(1)); + } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java b/common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java index ccc9204..852845e 100644 --- a/common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/BinaryExpression.java @@ -18,6 +18,9 @@ package org.apache.atlas.groovy; +import java.util.Arrays; +import java.util.List; + /** * Represents any kind of binary expression. This could * be an arithmetic expression, such as a + 3, a boolean @@ -30,7 +33,7 @@ public abstract class BinaryExpression extends AbstractGroovyExpression { private GroovyExpression left; private GroovyExpression right; - private String op; + protected String op; public BinaryExpression(GroovyExpression left, String op, GroovyExpression right) { this.left = left; @@ -48,4 +51,8 @@ public abstract class BinaryExpression extends AbstractGroovyExpression { right.generateGroovy(context); } + @Override + public List getChildren() { + return Arrays.asList(left, right); + } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/CastExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/CastExpression.java b/common/src/main/java/org/apache/atlas/groovy/CastExpression.java index 963724c..808f327 100644 --- a/common/src/main/java/org/apache/atlas/groovy/CastExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/CastExpression.java @@ -18,6 +18,9 @@ package org.apache.atlas.groovy; +import java.util.Collections; +import java.util.List; + /** * Groovy expression that represents a cast. */ @@ -28,7 +31,7 @@ public class CastExpression extends AbstractGroovyExpression { public CastExpression(GroovyExpression expr, String className) { this.expr = expr; - this.className =className; + this.className = className; } @Override @@ -41,4 +44,13 @@ public class CastExpression extends AbstractGroovyExpression { context.append(")"); } + @Override + public List getChildren() { + return Collections.singletonList(expr); + } + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.size() == 1; + return new CastExpression(newChildren.get(0), className); + } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java b/common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java index 2d70209..a5ca0b6 100644 --- a/common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/ClosureExpression.java @@ -20,6 +20,7 @@ package org.apache.atlas.groovy; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.Iterator; import java.util.List; @@ -28,28 +29,79 @@ import java.util.List; */ public class ClosureExpression extends AbstractGroovyExpression { - private List varNames = new ArrayList<>(); - private GroovyExpression body; + /** + * Variable declaration in a closure. + */ + public static class VariableDeclaration { + private String type; + private String varName; - public ClosureExpression(GroovyExpression body, String... varNames) { - this.body = body; - this.varNames.addAll(Arrays.asList(varNames)); + public VariableDeclaration(String type, String varName) { + super(); + this.type = type; + this.varName = varName; + } + + public VariableDeclaration(String varName) { + this.varName = varName; + } + + public void append(GroovyGenerationContext context) { + if (type != null) { + context.append(type); + context.append(" "); + } + context.append(varName); + } + } + private List vars = new ArrayList<>(); + private StatementListExpression body = new StatementListExpression(); + + public ClosureExpression(String... varNames) { + this(null, varNames); + } + + public ClosureExpression(GroovyExpression initialStmt, String... varNames) { + this(Arrays.asList(varNames), initialStmt); + } + + public ClosureExpression(List varNames, GroovyExpression initialStmt) { + if (initialStmt != null) { + this.body.addStatement(initialStmt); + } + for (String varName : varNames) { + vars.add(new VariableDeclaration(varName)); + } } - public ClosureExpression(List varNames, GroovyExpression body) { - this.body = body; - this.varNames.addAll(varNames); + public ClosureExpression(GroovyExpression initialStmt, List varNames) { + if (initialStmt != null) { + this.body.addStatement(initialStmt); + } + vars.addAll(varNames); + } + + public void addStatement(GroovyExpression expr) { + body.addStatement(expr); + } + + public void addStatements(List exprs) { + body.addStatements(exprs); + } + + public void replaceStatement(int index, GroovyExpression newExpr) { + body.replaceStatement(index, newExpr); } @Override public void generateGroovy(GroovyGenerationContext context) { context.append("{"); - if (!varNames.isEmpty()) { - Iterator varIt = varNames.iterator(); + if (!vars.isEmpty()) { + Iterator varIt = vars.iterator(); while(varIt.hasNext()) { - String varName = varIt.next(); - context.append(varName); + VariableDeclaration var = varIt.next(); + var.append(context); if (varIt.hasNext()) { context.append(", "); } @@ -58,6 +110,20 @@ public class ClosureExpression extends AbstractGroovyExpression { } body.generateGroovy(context); context.append("}"); + } + + @Override + public List getChildren() { + return Collections.singletonList(body); + } + public List getStatements() { + return body.getStatements(); + } + + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.size() == 1; + return new ClosureExpression(newChildren.get(0), vars); } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java b/common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java deleted file mode 100644 index 9a726f2..0000000 --- a/common/src/main/java/org/apache/atlas/groovy/CodeBlockExpression.java +++ /dev/null @@ -1,61 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.atlas.groovy; - -import java.util.ArrayList; -import java.util.Iterator; -import java.util.List; - -/** - * Groovy expression that represents a block of code - * that contains 0 or more statements that are delimited - * by semicolons. - */ -public class CodeBlockExpression extends AbstractGroovyExpression { - - private List body = new ArrayList<>(); - - public void addStatement(GroovyExpression expr) { - body.add(expr); - } - - public void addStatements(List exprs) { - body.addAll(exprs); - } - - @Override - public void generateGroovy(GroovyGenerationContext context) { - - /* - * the L:{} represents a groovy code block; the label is needed - * to distinguish it from a groovy closure. - */ - context.append("L:{"); - Iterator stmtIt = body.iterator(); - while(stmtIt.hasNext()) { - GroovyExpression stmt = stmtIt.next(); - stmt.generateGroovy(context); - if (stmtIt.hasNext()) { - context.append(";"); - } - } - context.append("}"); - - } -} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java b/common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java index 345f838..b64533f 100644 --- a/common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/ComparisonExpression.java @@ -17,6 +17,8 @@ */ package org.apache.atlas.groovy; +import java.util.List; + import org.apache.atlas.AtlasException; /** @@ -61,4 +63,14 @@ public class ComparisonExpression extends BinaryExpression { public ComparisonExpression(GroovyExpression left, ComparisonOperator op, GroovyExpression right) { super(left, op.getGroovyValue(), right); } + + private ComparisonExpression(GroovyExpression left, String op, GroovyExpression right) { + super(left, op, right); + } + + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.size() == 2; + return new ComparisonExpression(newChildren.get(0), op, newChildren.get(1)); + } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java b/common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java index 63638b7..c9e363e 100644 --- a/common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/ComparisonOperatorExpression.java @@ -17,6 +17,8 @@ */ package org.apache.atlas.groovy; +import java.util.List; + /** * Represents an expression that compares two expressions using * the Groovy "spaceship" operator. This is basically the @@ -29,4 +31,10 @@ public class ComparisonOperatorExpression extends BinaryExpression { public ComparisonOperatorExpression(GroovyExpression left, GroovyExpression right) { super(left, "<=>", right); } + + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.size() == 2; + return new ComparisonOperatorExpression(newChildren.get(0), newChildren.get(1)); + } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/FieldExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/FieldExpression.java b/common/src/main/java/org/apache/atlas/groovy/FieldExpression.java index f6d06bd..6a182ad 100644 --- a/common/src/main/java/org/apache/atlas/groovy/FieldExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/FieldExpression.java @@ -18,27 +18,38 @@ package org.apache.atlas.groovy; +import java.util.Collections; +import java.util.List; + /** * Groovy expression that accesses a field in an object. */ -public class FieldExpression extends AbstractGroovyExpression { +public class FieldExpression extends AbstractFunctionExpression { - private GroovyExpression target; private String fieldName; public FieldExpression(GroovyExpression target, String fieldName) { - this.target = target; + super(target); this.fieldName = fieldName; } @Override public void generateGroovy(GroovyGenerationContext context) { - - target.generateGroovy(context); + getCaller().generateGroovy(context); context.append(".'"); context.append(fieldName); context.append("'"); } + @Override + public List getChildren() { + return Collections.singletonList(getCaller()); + } + + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.size() == 1; + return new FieldExpression(newChildren.get(0), fieldName); + } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java b/common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java index dd9b1d5..ad09e3f 100644 --- a/common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/FunctionCallExpression.java @@ -20,41 +20,52 @@ package org.apache.atlas.groovy; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.Iterator; import java.util.List; /** * Groovy expression that calls a method on an object. */ -public class FunctionCallExpression extends AbstractGroovyExpression { - - // null for global functions - private GroovyExpression target; +public class FunctionCallExpression extends AbstractFunctionExpression { private String functionName; private List arguments = new ArrayList<>(); - public FunctionCallExpression(String functionName, List arguments) { - this.target = null; + public FunctionCallExpression(TraversalStepType type, String functionName, GroovyExpression... arguments) { + super(type, null); this.functionName = functionName; - this.arguments.addAll(arguments); + this.arguments.addAll(Arrays.asList(arguments)); } - public FunctionCallExpression(GroovyExpression target, String functionName, - List arguments) { - this.target = target; + public FunctionCallExpression(String functionName, GroovyExpression... arguments) { + super(null); + this.functionName = functionName; + this.arguments.addAll(Arrays.asList(arguments)); + } + + public FunctionCallExpression(TraversalStepType type, String functionName, List arguments) { + super(type, null); this.functionName = functionName; this.arguments.addAll(arguments); } - public FunctionCallExpression(String functionName, GroovyExpression... arguments) { - this.target = null; + public FunctionCallExpression(GroovyExpression target, String functionName, GroovyExpression... arguments) { + super(target); this.functionName = functionName; this.arguments.addAll(Arrays.asList(arguments)); } - public FunctionCallExpression(GroovyExpression target, String functionName, GroovyExpression... arguments) { - this.target = target; + public FunctionCallExpression(TraversalStepType type, GroovyExpression target, String functionName, + List arguments) { + super(type, target); + this.functionName = functionName; + this.arguments.addAll(arguments); + } + + public FunctionCallExpression(TraversalStepType type, GroovyExpression target, String functionName, + GroovyExpression... arguments) { + super(type, target); this.functionName = functionName; this.arguments.addAll(Arrays.asList(arguments)); } @@ -63,11 +74,20 @@ public class FunctionCallExpression extends AbstractGroovyExpression { arguments.add(expr); } + public List getArguments() { + return Collections.unmodifiableList(arguments); + } + + + public String getFunctionName() { + return functionName; + } + @Override public void generateGroovy(GroovyGenerationContext context) { - if (target != null) { - target.generateGroovy(context); + if (getCaller() != null) { + getCaller().generateGroovy(context); context.append("."); } context.append(functionName); @@ -77,10 +97,44 @@ public class FunctionCallExpression extends AbstractGroovyExpression { GroovyExpression expr = it.next(); expr.generateGroovy(context); if (it.hasNext()) { - context.append(", "); + context.append(","); } } context.append(")"); } + @Override + public List getChildren() { + List result = new ArrayList<>(arguments.size() + 1); + if (getCaller() != null) { + result.add(getCaller()); + } + result.addAll(arguments); + return result; + } + + @Override + public GroovyExpression copy(List newChildren) { + + if (getCaller() == null) { + return new FunctionCallExpression(getType(), functionName, newChildren); + } + + GroovyExpression newTarget = newChildren.get(0); + List args = null; + if (newChildren.size() > 1) { + args = newChildren.subList(1, newChildren.size()); + } else { + args = Collections.emptyList(); + } + return new FunctionCallExpression(getType(), newTarget, functionName, args); + + } + + public void setArgument(int index, GroovyExpression value) { + if (index < 0 || index >= arguments.size()) { + throw new IllegalArgumentException("Invalid argIndex " + index); + } + arguments.set(index, value); + } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java b/common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java index 493bd3d..8399147 100644 --- a/common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/GroovyExpression.java @@ -18,17 +18,55 @@ package org.apache.atlas.groovy; +import java.util.List; + /** * Represents an expression in the Groovy programming language, which * is the language that Gremlin scripts are written and interpreted in. */ public interface GroovyExpression { - /** - * Generates a groovy script from the expression. + * Generates a Groovy script from the expression. * * @param context */ void generateGroovy(GroovyGenerationContext context); + /** + * Gets all of the child expressions of this expression. + * s + * @return + */ + List getChildren(); + + /** + * Makes a copy of the expression, keeping everything the + * same except its child expressions. These are replaced + * with the provided children. The order of the children + * is important. It is expected that the children provided + * here are updated versions of the children returned by + * getChildren(). The order of the children must be the + * same as the order in which the children were returned + * by getChildren() + * + * @param newChildren + * @return + */ + GroovyExpression copy(List newChildren); + + /** + * Makes a shallow copy of the GroovyExpression. This + * is equivalent to copy(getChildren()); + * + * @return + */ + GroovyExpression copy(); + + /** + * Gets the type of traversal step represented by this + * expression (or TraversalStepType.NONE if it is not part of a graph traversal). + * + * @return + */ + TraversalStepType getType(); } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java b/common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java index 6abdbf0..4c0694a 100644 --- a/common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/IdentifierExpression.java @@ -18,18 +18,28 @@ package org.apache.atlas.groovy; +import java.util.Collections; +import java.util.List; + /** * Groovy expression that references the variable with the given name. * */ public class IdentifierExpression extends AbstractGroovyExpression { + private TraversalStepType type = TraversalStepType.NONE; private String varName; public IdentifierExpression(String varName) { this.varName = varName; } + public IdentifierExpression(TraversalStepType type, String varName) { + this.varName = varName; + this.type = type; + } + + public String getVariableName() { return varName; } @@ -39,4 +49,25 @@ public class IdentifierExpression extends AbstractGroovyExpression { context.append(varName); } + @Override + public List getChildren() { + return Collections.emptyList(); + } + + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.isEmpty(); + IdentifierExpression result = new IdentifierExpression(varName); + result.setType(type); + return result; + } + + public void setType(TraversalStepType type) { + this.type = type; + } + + @Override + public TraversalStepType getType() { + return type; + } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/LabeledExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/LabeledExpression.java b/common/src/main/java/org/apache/atlas/groovy/LabeledExpression.java new file mode 100644 index 0000000..a993410 --- /dev/null +++ b/common/src/main/java/org/apache/atlas/groovy/LabeledExpression.java @@ -0,0 +1,54 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.atlas.groovy; + +import java.util.Collections; +import java.util.List; + +/** + * Represents a Groovy expression that has a label. + */ +public class LabeledExpression extends AbstractGroovyExpression { + + private String label; + private GroovyExpression expr; + + public LabeledExpression(String label, GroovyExpression expr) { + this.label = label; + this.expr = expr; + } + + @Override + public void generateGroovy(GroovyGenerationContext context) { + context.append(label); + context.append(":"); + expr.generateGroovy(context); + } + + @Override + public List getChildren() { + return Collections.singletonList(expr); + } + + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.size() == 1; + return new LabeledExpression(label, newChildren.get(0)); + } +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/ListExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/ListExpression.java b/common/src/main/java/org/apache/atlas/groovy/ListExpression.java index f7acaac..7969426 100644 --- a/common/src/main/java/org/apache/atlas/groovy/ListExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/ListExpression.java @@ -20,6 +20,7 @@ package org.apache.atlas.groovy; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.Iterator; import java.util.List; @@ -57,4 +58,15 @@ public class ListExpression extends AbstractGroovyExpression { context.append("]"); } + @Override + public List getChildren() { + return Collections.unmodifiableList(values); + } + + @Override + public GroovyExpression copy(List newChildren) { + return new ListExpression(newChildren); + } + + } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java b/common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java index 008c885..1407499 100644 --- a/common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java @@ -18,13 +18,15 @@ package org.apache.atlas.groovy; +import java.util.Collections; +import java.util.List; import java.util.regex.Matcher; import java.util.regex.Pattern; /** * Represents a literal value. */ -public class LiteralExpression implements GroovyExpression { +public class LiteralExpression extends AbstractGroovyExpression { public static final LiteralExpression TRUE = new LiteralExpression(true); public static final LiteralExpression FALSE = new LiteralExpression(false); @@ -40,6 +42,12 @@ public class LiteralExpression implements GroovyExpression { this.addTypeSuffix = addTypeSuffix; } + public LiteralExpression(Object value, boolean addTypeSuffix, boolean translateToParameter) { + this.value = value; + this.translateToParameter = translateToParameter; + this.addTypeSuffix = addTypeSuffix; + } + public LiteralExpression(Object value) { this.value = value; this.translateToParameter = value instanceof String; @@ -86,6 +94,10 @@ public class LiteralExpression implements GroovyExpression { } + public Object getValue() { + return value; + } + private String getEscapedValue() { String escapedValue = (String)value; escapedValue = escapedValue.replaceAll(Pattern.quote("\\"), Matcher.quoteReplacement("\\\\")); @@ -96,4 +108,15 @@ public class LiteralExpression implements GroovyExpression { public void setTranslateToParameter(boolean translateToParameter) { this.translateToParameter = translateToParameter; } + + @Override + public List getChildren() { + return Collections.emptyList(); + } + + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.size() == 0; + return new LiteralExpression(value, addTypeSuffix, translateToParameter); + } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java b/common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java index ee5829b..68e6847 100644 --- a/common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/LogicalExpression.java @@ -17,6 +17,8 @@ */ package org.apache.atlas.groovy; +import java.util.List; + /** * Represents a logical (and/or) expression. * @@ -43,4 +45,14 @@ public class LogicalExpression extends BinaryExpression { public LogicalExpression(GroovyExpression left, LogicalOperator op, GroovyExpression right) { super(left, op.getGroovyValue(), right); } + + private LogicalExpression(GroovyExpression left, String op, GroovyExpression right) { + super(left, op, right); + } + + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.size() == 2; + return new LogicalExpression(newChildren.get(0), op, newChildren.get(1)); + } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/RangeExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/RangeExpression.java b/common/src/main/java/org/apache/atlas/groovy/RangeExpression.java index 7322f69..977adb6 100644 --- a/common/src/main/java/org/apache/atlas/groovy/RangeExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/RangeExpression.java @@ -18,28 +18,68 @@ package org.apache.atlas.groovy; +import java.util.Collections; +import java.util.List; + /** * Represents an "exclusive" range expression, e.g. [0..<10]. */ -public class RangeExpression extends AbstractGroovyExpression { +public class RangeExpression extends AbstractFunctionExpression { - private GroovyExpression parent; - private int offset; - private int count; + private TraversalStepType stepType; + private int startIndex; + private int endIndex; - public RangeExpression(GroovyExpression parent, int offset, int count) { - this.parent = parent; - this.offset = offset; - this.count = count; + public RangeExpression(TraversalStepType stepType, GroovyExpression parent, int offset, int count) { + super(parent); + this.startIndex = offset; + this.endIndex = count; + this.stepType = stepType; } @Override public void generateGroovy(GroovyGenerationContext context) { - parent.generateGroovy(context); + getCaller().generateGroovy(context); context.append(" ["); - new LiteralExpression(offset).generateGroovy(context); + new LiteralExpression(startIndex).generateGroovy(context); context.append("..<"); - new LiteralExpression(count).generateGroovy(context); + new LiteralExpression(endIndex).generateGroovy(context); context.append("]"); } + + @Override + public List getChildren() { + return Collections.singletonList(getCaller()); + } + + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.size() == 1; + return new RangeExpression(stepType, newChildren.get(0), startIndex, endIndex); + } + + @Override + public TraversalStepType getType() { + return stepType; + } + + public int getStartIndex() { + + return startIndex; + } + + public void setStartIndex(int startIndex) { + + this.startIndex = startIndex; + } + + public int getEndIndex() { + + return endIndex; + } + + public void setEndIndex(int endIndex) { + + this.endIndex = endIndex; + } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java b/common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java new file mode 100644 index 0000000..f9c88ec --- /dev/null +++ b/common/src/main/java/org/apache/atlas/groovy/StatementListExpression.java @@ -0,0 +1,98 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.atlas.groovy; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.Iterator; +import java.util.List; + +/** + * Represents a semi-colon delimited list of Groovy expressions. + */ +public class StatementListExpression extends AbstractGroovyExpression { + + private List stmts = new ArrayList<>(); + + public StatementListExpression() { + + } + + /** + * @param newChildren + */ + public StatementListExpression(List newChildren) { + stmts.addAll(newChildren); + } + + public void addStatement(GroovyExpression expr) { + if (expr instanceof StatementListExpression) { + stmts.addAll(((StatementListExpression)expr).getStatements()); + } else { + stmts.add(expr); + } + } + + public void addStatements(List exprs) { + for(GroovyExpression expr : exprs) { + addStatement(expr); + } + } + + @Override + public void generateGroovy(GroovyGenerationContext context) { + + Iterator stmtIt = stmts.iterator(); + while(stmtIt.hasNext()) { + GroovyExpression stmt = stmtIt.next(); + stmt.generateGroovy(context); + if (stmtIt.hasNext()) { + context.append(";"); + } + } + } + + + public List getStatements() { + return stmts; + } + + @Override + public List getChildren() { + return Collections.unmodifiableList(stmts); + } + + @Override + public GroovyExpression copy(List newChildren) { + return new StatementListExpression(newChildren); + } + + @Override + public TraversalStepType getType() { + return TraversalStepType.NONE; + } + + /** + * @param oldExpr + * @param newExpr + */ + public void replaceStatement(int index, GroovyExpression newExpr) { + stmts.set(index, newExpr); + } +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java b/common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java index 75a2f86..8835dd2 100644 --- a/common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/TernaryOperatorExpression.java @@ -18,6 +18,9 @@ package org.apache.atlas.groovy; +import java.util.Arrays; +import java.util.List; + /** * Groovy expression that represents the ternary operator (expr ? trueValue : * falseValue) @@ -29,7 +32,7 @@ public class TernaryOperatorExpression extends AbstractGroovyExpression { private GroovyExpression falseValue; public TernaryOperatorExpression(GroovyExpression booleanExpr, GroovyExpression trueValue, - GroovyExpression falseValue) { + GroovyExpression falseValue) { this.booleanExpr = booleanExpr; this.trueValue = trueValue; @@ -41,9 +44,9 @@ public class TernaryOperatorExpression extends AbstractGroovyExpression { context.append("(("); booleanExpr.generateGroovy(context); - context.append(") ? ("); + context.append(")?("); trueValue.generateGroovy(context); - context.append(") : ("); + context.append("):("); falseValue.generateGroovy(context); context.append("))"); } @@ -53,4 +56,20 @@ public class TernaryOperatorExpression extends AbstractGroovyExpression { generateGroovy(context); return context.getQuery(); } + + @Override + public List getChildren() { + return Arrays.asList(booleanExpr, trueValue, falseValue); + } + + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.size() == 3; + return new TernaryOperatorExpression(newChildren.get(0), newChildren.get(1), newChildren.get(2)); + } + + @Override + public TraversalStepType getType() { + return trueValue.getType(); + } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java b/common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java new file mode 100644 index 0000000..fde8814 --- /dev/null +++ b/common/src/main/java/org/apache/atlas/groovy/TraversalStepType.java @@ -0,0 +1,121 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.groovy; + +/** + * Types of graph traversal steps. These are based on the traversal steps + * described in the TinkerPop documentation at + * http://tinkerpop.apache.org/docs/current/reference/#graph-traversal-steps. + */ +public enum TraversalStepType { + /** + * Indicates that the expression is not part of a graph traversal. + */ + NONE, + + /** + * Indicates that the expression is a + * {@link org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource}. + * This is not technically a graph traversal step. This is the expression the traversal is started from ("g"). + */ + SOURCE, + + /** + * A Start step adds vertices or edges to the traversal. These include "V", "E", and "inject". + */ + START, + + /** + * An End step causes the traversal to be executed. This includes steps such as "toList", "toSet", and "fill" + */ + END, + + /** + * Map steps map the current traverser value to exactly one new value. These + * steps include "map" and "select". Here, we make a further distinction + * based on the type of expression that things are being mapped to. + *

+ * MAP_TO_ELEMENT indicates that the traverser value is being mapped + * to either a Vertex or an Edge. + */ + MAP_TO_ELEMENT, + /** + * Map steps map the current traverser value to exactly one new value. These + * steps include "map" and "select". Here, we make a further distinction + * based on the type of expression that things are being mapped to. + *

+ * MAP_TO_VALUE indicates that the traverser value is being mapped + * to something that is not a Vertex or an Edge. + */ + MAP_TO_VALUE, + + /** + * FlatMap steps map the current value of the traverser to an iterator of objects that + * are streamed to the next step. These are steps like "in, "out", "inE", and + * so forth which map the current value of the traverser from some vertex or edge + * to some other set of vertices or edges that is derived from the original set based + * on the structure of the graph. This also includes "values", which maps a vertex or + * edge to the set of values for a given property. Here, we make a further distinction + * based on the type of expression that things are being mapped to. + *

+ * FLAT_MAP_TO_ELEMENTS indicates that the traverser value is being mapped + * to something that is a Vertex or an Edge (in, out, outE fall in this category). + */ + FLAT_MAP_TO_ELEMENTS, + + /** + * FlatMap steps map the current value of the traverser to an iterator of objects that + * are streamed to the next step. These are steps like "in, "out", "inE", and + * so forth which map the current value of the traverser from some vertex or edge + * to some other set of vertices or edges that is derived from the original set based + * on the structure of the graph. This also includes "values", which maps a vertex or + * edge to the set of values for a given property. Here, we make a further distinction + * based on the type of expression that things are being mapped to. + *

+ * FLAT_MAP_TO_VALUES indicates that the traverser value is being mapped + * to something that not is a Vertex or an Edge (values falls in this category). + */ + FLAT_MAP_TO_VALUES, + + /** + * Filter steps filter things out of the traversal. These include "has", "where", + * "and", "or", and "filter". + */ + FILTER, + + /** + * Side effect steps do not affect the traverser value, but do something + * that affects the state of the traverser. These include things such as + * "enablePath()", "as", and "by". + */ + SIDE_EFFECT, + + /** + * Branch steps split the traverser, for example, "repeat", "branch", "choose", and "union". + */ + BRANCH, + + /** + * Barrier steps in Gremlin force everything before them to be executed + * before moving on to the steps after them. We also use this to indicate + * steps that need to do some aggregation or processing that requires the + * full query result to be present in order for the step to work correctly. + * This includes "range", "group", and "order", and "cap" + */ + BARRIER, +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java b/common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java index 4a61052..956dafa 100644 --- a/common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/TypeCoersionExpression.java @@ -18,6 +18,9 @@ package org.apache.atlas.groovy; +import java.util.Collections; +import java.util.List; + /** * Groovy expression that represents a type coersion (e.g obj as Set). */ @@ -28,17 +31,29 @@ public class TypeCoersionExpression extends AbstractGroovyExpression { public TypeCoersionExpression(GroovyExpression expr, String className) { this.expr = expr; - this.className =className; + this.className = className; } @Override public void generateGroovy(GroovyGenerationContext context) { - context.append("("); + context.append("(("); expr.generateGroovy(context); context.append(")"); context.append(" as "); context.append(className); + context.append(")"); + } + + @Override + public List getChildren() { + return Collections.singletonList(expr); + } + + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.size() == 1; + return new TypeCoersionExpression(newChildren.get(0), className); } } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java b/common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java index 7e018f1..1aa7443 100644 --- a/common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java +++ b/common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java @@ -18,6 +18,9 @@ package org.apache.atlas.groovy; +import java.util.Collections; +import java.util.List; + /** * Groovy statement that assigns a value to a variable. */ @@ -50,9 +53,20 @@ public class VariableAssignmentExpression extends AbstractGroovyExpression { context.append(" "); } context.append(name); - context.append(" = "); + context.append("="); value.generateGroovy(context); } + @Override + public List getChildren() { + return Collections.singletonList(value); + } + + @Override + public GroovyExpression copy(List newChildren) { + assert newChildren.size() == 1; + return new VariableAssignmentExpression(name, newChildren.get(0)); + } + } http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/distro/src/conf/atlas-application.properties ---------------------------------------------------------------------- diff --git a/distro/src/conf/atlas-application.properties b/distro/src/conf/atlas-application.properties index d9e2f6e..3e71a26 100755 --- a/distro/src/conf/atlas-application.properties +++ b/distro/src/conf/atlas-application.properties @@ -29,6 +29,13 @@ atlas.graph.storage.hbase.table=apache_atlas_titan ${titan.storage.properties} +# Gremlin Query Optimizer +# +# Enables rewriting gremlin queries to maximize performance. This flag is provided as +# a possible way to work around any defects that are found in the optimizer until they +# are resolved. +#atlas.query.gremlinOptimizerEnabled=true + # Delete handler # # This allows the default behavior of doing "soft" deletes to be changed. http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/graphdb/titan0/pom.xml ---------------------------------------------------------------------- diff --git a/graphdb/titan0/pom.xml b/graphdb/titan0/pom.xml index 58a5cb8..9d88a72 100644 --- a/graphdb/titan0/pom.xml +++ b/graphdb/titan0/pom.xml @@ -34,6 +34,7 @@ 2.6.0 0.5.4 + 14.0 @@ -53,6 +54,13 @@ + com.google.guava + guava + ${guava.version} + + + + com.google.inject guice provided http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/intg/pom.xml ---------------------------------------------------------------------- diff --git a/intg/pom.xml b/intg/pom.xml index 52b5ef5..a5fab71 100644 --- a/intg/pom.xml +++ b/intg/pom.xml @@ -60,7 +60,6 @@ com.google.guava guava - ${guava.version} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index a985792..834ecae 100644 --- a/pom.xml +++ b/pom.xml @@ -459,7 +459,7 @@ 3.1.3.RELEASE 1.3.1.RELEASE 3.1.0 - 18.0 + 19.0 1.0 @@ -633,6 +633,12 @@ + + + com.google.guava + guava + ${guava.version} + org.aspectj http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/release-log.txt ---------------------------------------------------------------------- diff --git a/release-log.txt b/release-log.txt index fa22e6d..b0fc288 100644 --- a/release-log.txt +++ b/release-log.txt @@ -9,8 +9,9 @@ ATLAS-1060 Add composite indexes for exact match performance improvements for al ATLAS-1127 Modify creation and modification timestamps to Date instead of Long(sumasai) ALL CHANGES: -ATLAS-1513: updated AtlasEntityType with methods to get foreign-key references; added helper methods in AtlasAttribute (mneethiraj via kevalbhatt) -ATLAS-1502: added configuration to restrict entity-types editable via UI (Kalyanikashikar via mneethiraj) +ATLAS-1369 Optimize Gremlin queries generated by DSL translator (jnhagelb) +ATLAS-1513 updated AtlasEntityType with methods to get foreign-key references; added helper methods in AtlasAttribute (mneethiraj via kevalbhatt) +ATLAS-1502 added configuration to restrict entity-types editable via UI (Kalyanikashikar via mneethiraj) ATLAS-1507 fixed incorrect relationship specified in hive-model ATLAS-1506 updated AtlasObjectId to support unqiueAttributes to identity the object ATLAS-1378 Use .gitignore so git does not see binary files as changed (david_radley via dkantor) http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java ---------------------------------------------------------------------- diff --git a/repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java b/repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java index 4db4773..8fb9ddd 100644 --- a/repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java +++ b/repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java @@ -25,6 +25,7 @@ import org.apache.atlas.AtlasException; import org.apache.atlas.GraphTransaction; import org.apache.atlas.discovery.graph.DefaultGraphPersistenceStrategy; import org.apache.atlas.discovery.graph.GraphBackedDiscoveryService; +import org.apache.atlas.query.GremlinQueryResult; import org.apache.atlas.query.InputLineageClosureQuery; import org.apache.atlas.query.OutputLineageClosureQuery; import org.apache.atlas.query.QueryParams; @@ -139,7 +140,8 @@ public class DataSetLineageService implements LineageService { guid, HIVE_PROCESS_TYPE_NAME, HIVE_PROCESS_INPUT_ATTRIBUTE_NAME, HIVE_PROCESS_OUTPUT_ATTRIBUTE_NAME, Option.empty(), SELECT_ATTRIBUTES, true, graphPersistenceStrategy, graph); - return inputsQuery.graph(null).toInstanceJson(); + GremlinQueryResult result = inputsQuery.evaluate(); + return inputsQuery.graph(result).toInstanceJson(); } @Override @@ -156,7 +158,8 @@ public class DataSetLineageService implements LineageService { new OutputLineageClosureQuery(AtlasClient.DATA_SET_SUPER_TYPE, SELECT_INSTANCE_GUID, guid, HIVE_PROCESS_TYPE_NAME, HIVE_PROCESS_INPUT_ATTRIBUTE_NAME, HIVE_PROCESS_OUTPUT_ATTRIBUTE_NAME, Option.empty(), SELECT_ATTRIBUTES, true, graphPersistenceStrategy, graph); - return outputsQuery.graph(null).toInstanceJson(); + GremlinQueryResult result = outputsQuery.evaluate(); + return outputsQuery.graph(result).toInstanceJson(); } /** http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/aa74c73d/repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java ---------------------------------------------------------------------- diff --git a/repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java b/repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java index 1858739..798d909 100644 --- a/repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java +++ b/repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java @@ -18,7 +18,12 @@ package org.apache.atlas.gremlin; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + import org.apache.atlas.AtlasException; +import org.apache.atlas.groovy.AbstractFunctionExpression; import org.apache.atlas.groovy.CastExpression; import org.apache.atlas.groovy.ClosureExpression; import org.apache.atlas.groovy.ComparisonExpression; @@ -34,13 +39,11 @@ import org.apache.atlas.groovy.LogicalExpression; import org.apache.atlas.groovy.LogicalExpression.LogicalOperator; import org.apache.atlas.groovy.RangeExpression; import org.apache.atlas.groovy.TernaryOperatorExpression; +import org.apache.atlas.groovy.TraversalStepType; import org.apache.atlas.query.GraphPersistenceStrategies; import org.apache.atlas.query.TypeUtils.FieldInfo; import org.apache.atlas.typesystem.types.IDataType; -import java.util.ArrayList; -import java.util.List; - /** * Generates gremlin query expressions using Gremlin 2 syntax. @@ -54,12 +57,11 @@ public class Gremlin2ExpressionFactory extends GremlinExpressionFactory { private static final String PATH_FIELD = "path"; private static final String ENABLE_PATH_METHOD = "enablePath"; private static final String BACK_METHOD = "back"; - private static final String VERTEX_LIST_CLASS = "List"; - private static final String VERTEX_ARRAY_CLASS = "Vertex[]"; private static final String LAST_METHOD = "last"; + @Override public GroovyExpression generateLogicalExpression(GroovyExpression parent, String operator, List operands) { - return new FunctionCallExpression(parent, operator, operands); + return new FunctionCallExpression(TraversalStepType.FILTER, parent, operator, operands); } @@ -72,7 +74,7 @@ public class Gremlin2ExpressionFactory extends GremlinExpressionFactory { return parent; } else { - return new FunctionCallExpression(parent, BACK_METHOD, new LiteralExpression(alias)); + return new FunctionCallExpression(TraversalStepType.MAP_TO_ELEMENT, parent, BACK_METHOD, new LiteralExpression(alias)); } } @@ -100,23 +102,23 @@ public class Gremlin2ExpressionFactory extends GremlinExpressionFactory { whileFunction = new ClosureExpression(new TernaryOperatorExpression(pathContainsExpr, LiteralExpression.FALSE, LiteralExpression.TRUE)); } GroovyExpression emitFunction = new ClosureExpression(emitExpr); - GroovyExpression loopCall = new FunctionCallExpression(loopExpr, LOOP_METHOD, new LiteralExpression(alias), whileFunction, emitFunction); + GroovyExpression loopCall = new FunctionCallExpression(TraversalStepType.BRANCH, loopExpr, LOOP_METHOD, new LiteralExpression(alias), whileFunction, emitFunction); - return new FunctionCallExpression(loopCall, ENABLE_PATH_METHOD); + return new FunctionCallExpression(TraversalStepType.SIDE_EFFECT, loopCall, ENABLE_PATH_METHOD); } @Override public GroovyExpression typeTestExpression(GraphPersistenceStrategies s, String typeName, GroovyExpression itRef) { - GroovyExpression typeAttrExpr = new FieldExpression(itRef, s.typeAttributeName()); GroovyExpression superTypeAttrExpr = new FieldExpression(itRef, s.superTypeAttributeName()); GroovyExpression typeNameExpr = new LiteralExpression(typeName); - - GroovyExpression typeMatchesExpr = new ComparisonExpression(typeAttrExpr, ComparisonOperator.EQUALS, typeNameExpr); GroovyExpression isSuperTypeExpr = new FunctionCallExpression(superTypeAttrExpr, CONTAINS, typeNameExpr); GroovyExpression superTypeMatchesExpr = new TernaryOperatorExpression(superTypeAttrExpr, isSuperTypeExpr, LiteralExpression.FALSE); + GroovyExpression typeAttrExpr = new FieldExpression(itRef, s.typeAttributeName()); + GroovyExpression typeMatchesExpr = new ComparisonExpression(typeAttrExpr, ComparisonOperator.EQUALS, typeNameExpr); return new LogicalExpression(typeMatchesExpr, LogicalOperator.OR, superTypeMatchesExpr); + } @Override @@ -129,7 +131,7 @@ public class Gremlin2ExpressionFactory extends GremlinExpressionFactory { for(GroovyExpression expr : srcExprs) { selectArgs.add(new ClosureExpression(expr)); } - return new FunctionCallExpression(parent, SELECT_METHOD, selectArgs); + return new FunctionCallExpression(TraversalStepType.MAP_TO_VALUE, parent, SELECT_METHOD, selectArgs); } @Override @@ -142,7 +144,7 @@ public class Gremlin2ExpressionFactory extends GremlinExpressionFactory { GroovyExpression requiredValue, FieldInfo fInfo) throws AtlasException { GroovyExpression op = gremlin2CompOp(symbol); GroovyExpression propertyNameExpr = new LiteralExpression(propertyName); - return new FunctionCallExpression(parent, HAS_METHOD, propertyNameExpr, op, requiredValue); + return new FunctionCallExpression(TraversalStepType.FILTER, parent, HAS_METHOD, propertyNameExpr, op, requiredValue); } private GroovyExpression gremlin2CompOp(String op) throws AtlasException { @@ -173,13 +175,52 @@ public class Gremlin2ExpressionFactory extends GremlinExpressionFactory { } @Override - protected GroovyExpression initialExpression(GraphPersistenceStrategies s, GroovyExpression varExpr) { - return new FunctionCallExpression(varExpr, "_"); + protected GroovyExpression initialExpression(GroovyExpression varExpr, GraphPersistenceStrategies s) { + return generateSeededTraversalExpresssion(false, varExpr); + } + + @Override + public GroovyExpression generateSeededTraversalExpresssion(boolean isMap, GroovyExpression varExpr) { + return new FunctionCallExpression(TraversalStepType.START, varExpr, "_"); } @Override - public GroovyExpression generateLimitExpression(GroovyExpression parent, int offset, int totalRows) { - return new RangeExpression(parent, offset, totalRows); + public GroovyExpression generateRangeExpression(GroovyExpression parent, int startIndex, int endIndex) { + //treat as barrier step, since limits need to be applied globally (even though it + //is technically a filter step) + return new RangeExpression(TraversalStepType.BARRIER, parent, startIndex, endIndex); + } + + @Override + public boolean isRangeExpression(GroovyExpression expr) { + + return (expr instanceof RangeExpression); + } + + @Override + public int[] getRangeParameters(AbstractFunctionExpression expr) { + + if (isRangeExpression(expr)) { + RangeExpression rangeExpression = (RangeExpression) expr; + return new int[] {rangeExpression.getStartIndex(), rangeExpression.getEndIndex()}; + } + else { + return null; + } + } + + @Override + public void setRangeParameters(GroovyExpression expr, int startIndex, int endIndex) { + + if (isRangeExpression(expr)) { + RangeExpression rangeExpression = (RangeExpression) expr; + rangeExpression.setStartIndex(startIndex); + rangeExpression.setEndIndex(endIndex); + } + else { + throw new IllegalArgumentException(expr.getClass().getName() + " is not a valid range expression - must be an instance of " + RangeExpression.class.getName()); + } + } @Override @@ -195,7 +236,7 @@ public class Gremlin2ExpressionFactory extends GremlinExpressionFactory { @Override public GroovyExpression generateOrderByExpression(GroovyExpression parent, List translatedOrderBy, boolean isAscending) { - GroovyExpression itExpr = getItVariable(); + GroovyExpression aPropertyExpr = translatedOrderBy.get(0); GroovyExpression bPropertyExpr = translatedOrderBy.get(1); @@ -212,27 +253,28 @@ public class Gremlin2ExpressionFactory extends GremlinExpressionFactory { else { comparisonFunction = new ComparisonOperatorExpression(bCondition, aCondition); } - return new FunctionCallExpression(parent, ORDER_METHOD, new ClosureExpression(comparisonFunction)); + return new FunctionCallExpression(TraversalStepType.BARRIER, parent, ORDER_METHOD, new ClosureExpression(comparisonFunction)); } + @Override public GroovyExpression getAnonymousTraversalExpression() { - return new FunctionCallExpression("_"); + return new FunctionCallExpression(TraversalStepType.START, "_"); } + + @Override public GroovyExpression generateGroupByExpression(GroovyExpression parent, GroovyExpression groupByExpression, - GroovyExpression aggregationFunction) { - + GroovyExpression aggregationFunction) { GroovyExpression groupByClosureExpr = new ClosureExpression(groupByExpression); GroovyExpression itClosure = new ClosureExpression(getItVariable()); - GroovyExpression result = new FunctionCallExpression(parent, "groupBy", groupByClosureExpr, itClosure); - result = new FunctionCallExpression(result, "cap"); - result = new FunctionCallExpression(result, "next"); + GroovyExpression result = new FunctionCallExpression(TraversalStepType.BARRIER, parent, "groupBy", groupByClosureExpr, itClosure); + result = new FunctionCallExpression(TraversalStepType.SIDE_EFFECT, result, "cap"); + result = new FunctionCallExpression(TraversalStepType.END, result, "next"); result = new FunctionCallExpression(result, "values"); result = new FunctionCallExpression(result, "toList"); - GroovyExpression mapValuesClosure = new ClosureExpression(getItVariable()); GroovyExpression aggregrationFunctionClosure = new ClosureExpression(aggregationFunction); result = new FunctionCallExpression(result, "collect", aggregrationFunctionClosure); return result; @@ -251,8 +293,49 @@ public class Gremlin2ExpressionFactory extends GremlinExpressionFactory { //assumes cast already performed @Override public GroovyExpression generateCountExpression(GroovyExpression itExpr) { - GroovyExpression collectionExpr = new CastExpression(itExpr,"Collection"); return new FunctionCallExpression(itExpr, "size"); } + + @Override + public String getTraversalExpressionClass() { + return "GremlinPipeline"; + } + + + @Override + public boolean isSelectGeneratesMap(int aliasCount) { + //in Gremlin 2 select always generates a map + return true; + } + + @Override + public GroovyExpression generateMapExpression(GroovyExpression parent, ClosureExpression closureExpression) { + return new FunctionCallExpression(TraversalStepType.MAP_TO_ELEMENT, parent, "transform", closureExpression); + } + + @Override + public GroovyExpression generateGetSelectedValueExpression(LiteralExpression key, + GroovyExpression rowMap) { + rowMap = new CastExpression(rowMap, "Row"); + GroovyExpression getExpr = new FunctionCallExpression(rowMap, "getColumn", key); + return getExpr; + } + + @Override + public GroovyExpression getCurrentTraverserObject(GroovyExpression traverser) { + return traverser; + } + + public List getAliasesRequiredByExpression(GroovyExpression expr) { + if(!(expr instanceof FunctionCallExpression)) { + return Collections.emptyList(); + } + FunctionCallExpression fc = (FunctionCallExpression)expr; + if(! fc.getFunctionName().equals(LOOP_METHOD)) { + return Collections.emptyList(); + } + LiteralExpression aliasName = (LiteralExpression)fc.getArguments().get(0); + return Collections.singletonList(aliasName.getValue().toString()); + } }