From issues-return-3516-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Sat Dec 22 03:41:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id E8285180627 for ; Sat, 22 Dec 2018 03:41:04 +0100 (CET) Received: (qmail 48324 invoked by uid 500); 22 Dec 2018 02:41:04 -0000 Mailing-List: contact issues-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list issues@phoenix.apache.org Received: (qmail 48315 invoked by uid 99); 22 Dec 2018 02:41:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Dec 2018 02:41:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 94D59C9639 for ; Sat, 22 Dec 2018 02:41:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id BHrikLKjtCqK for ; Sat, 22 Dec 2018 02:41:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 43D355F643 for ; Sat, 22 Dec 2018 02:41:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 609DAE00C9 for ; Sat, 22 Dec 2018 02:41:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 1C56025334 for ; Sat, 22 Dec 2018 02:41:00 +0000 (UTC) Date: Sat, 22 Dec 2018 02:41:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: issues@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (PHOENIX-4820) Optimize OrderBy for ClientAggregatePlan MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16727184#comment-16727184 ] ASF GitHub Bot commented on PHOENIX-4820: ----------------------------------------- Github user comnetwork commented on a diff in the pull request: https://github.com/apache/phoenix/pull/417#discussion_r243723430 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/expression/Expression.java --- @@ -88,4 +88,10 @@ * @return */ boolean requiresFinalEvaluation(); + --- End diff -- ExpressionUtil.isConstant(Expression) is not suitable for OrderPreservingTracker.IsConstantVisitor, because ExpressionUtil.isConstant(Expression) depends on Expression.isStateless() and Expression.getDeterminism(). Expression isStateless() is to check if the expression is depend on the server state, even for RowKeyColumnExpression , isStateless() is false. What we want to check is if a expression is constant when all children of it are constants, just consider following sql: select a.ak3 from (select rand() ak1,length(pk2) ak2,length(pk3) ak3,length(v1) av1,length(v2) av2 from test order by pk2,pk3 limit 10) a where a.ak1 = 0.0 and a.av2 = length(substr('abc',1,1)) group by a.ak3,CASE WHEN coalesce(a.ak1,1) > coalesce(a.av2,2) THEN coalesce(a.ak1,1) ELSE coalesce(a.av2,2) END,a.av1 order by a.ak3,a.av1 Obviously , because of rand(), the Determinism of expression a.ak1 is Determinism.PER_INVOCATION, so for expression "CASE WHEN coalesce(a.ak1,1) > coalesce(a.av2,2) THEN coalesce(a.ak1,1) ELSE coalesce(a.av2,2) END", the determinism is Determinism.PER_INVOCATION and isStateless is false , but because the a.ak1 and a.av2 are both constants in where clause of outer query, we can regard "CASE WHEN coalesce(a.ak1,1) > coalesce(a.av2,2) THEN coalesce(a.ak1,1) ELSE coalesce(a.av2,2) END" as constant in IsConstantVisitor. BTW. The value returned by ExpressionUtil.isConstant() is complicated and runtime related because of Expression.getDeterminism(). In following BaseCompoundExpression.init method, determinism is the combine of its children. ` private void init(List children) { this.children = ImmutableList.copyOf(children); boolean isStateless = true; boolean isNullable = false; boolean requiresFinalEvaluation = false; this.determinism = Determinism.ALWAYS; for (int i = 0; i < children.size(); i++) { Expression child = children.get(i); isNullable |= child.isNullable(); isStateless &= child.isStateless(); this.determinism = this.determinism.combine(child.getDeterminism()); requiresFinalEvaluation |= child.requiresFinalEvaluation(); } this.isStateless = isStateless; this.isNullable = isNullable; this.requiresFinalEvaluation = requiresFinalEvaluation; }` > Optimize OrderBy for ClientAggregatePlan > ---------------------------------------- > > Key: PHOENIX-4820 > URL: https://issues.apache.org/jira/browse/PHOENIX-4820 > Project: Phoenix > Issue Type: Improvement > Affects Versions: 4.14.0 > Reporter: chenglei > Assignee: chenglei > Priority: Major > Fix For: 4.15.0 > > Attachments: PHOENIX-4820-4.x-HBase-1.3.patch > > > Given a table > {code} > create table test ( > pk1 varchar not null , > pk2 varchar not null, > pk3 varchar not null, > v1 varchar, > v2 varchar, > CONSTRAINT TEST_PK PRIMARY KEY ( > pk1, > pk2, > pk3 )) > {code} > for following sql : > {code} > select a.ak3 > from (select substr(pk1,1,1) ak1,substr(pk2,1,1) ak2,substr(pk3,1,1) ak3,substr(v1,1,1) av1,substr(v2,1,1) av2 from test order by pk2,pk3 limit 10) a group by a.ak3,a.av1 order by a.ak3,a.av1 > {code} > Intuitively, the above OrderBy statement {{order by a.ak3,a.av1}} should be compiled out because it match the group by statement, but in fact it is not. > The problem is caused by the {{QueryCompiler.compileSingleQuery}} and {{QueryCompiler.compileSingleFlatQuery}},for {{QueryCompiler.compileSingleQuery}} method,because the inner query has order by, so in line 520, local variable {{isInRowKeyOrder}} is false: > {code} > 519 context.setCurrentTable(tableRef); > 520 boolean isInRowKeyOrder = innerPlan.getGroupBy() == GroupBy.EMPTY_GROUP_BY && innerPlan.getOrderBy() == OrderBy.EMPTY_ORDER_BY; > {code} > In {{QueryCompiler.compileSingleFlatQuery}},when {{OrderByCompiler.compile}} method is invoked, the last parameter {{isInRowKeyOrder}} is false: > {code} > 562 OrderBy orderBy = OrderByCompiler.compile(context, select, groupBy, limit, offset, projector, > 563 groupBy == GroupBy.EMPTY_GROUP_BY ? innerPlanTupleProjector : null, isInRowKeyOrder); > {code} > So in following line 156 for {{OrderByCompiler.compile}},even though the {{tracker.isOrderPreserving}} is true, the OrderBy statement could not be compiled out. > {code} > 156 if (isInRowKeyOrder && tracker.isOrderPreserving()) { > {code} > In my opinion, with GroupBy, in following line 563 for {{QueryCompiler.compileSingleFlatQuery}} method, when we call > {{OrderByCompiler.compile}} method, we no need to conside the {{isInRowKeyOrder}}, just like the previous parameter {{tupleProjector}} does. > {code} > 562 OrderBy orderBy = OrderByCompiler.compile(context, select, groupBy, limit, offset, projector, > 563 groupBy == GroupBy.EMPTY_GROUP_BY ? innerPlanTupleProjector : null, isInRowKeyOrder); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)