hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vitalii Diravka (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-19101) Apply rule [HiveJoinPushTransitivePredicatesRule] is getting stuck when there are huge number of predicates
Date Sat, 12 May 2018 14:22:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-19101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16473106#comment-16473106
] 

Vitalii Diravka commented on HIVE-19101:
----------------------------------------

[~jcamachorodriguez] I think moving onto Calcite JoinPushTransitivePredicatesRule and RelMdPredicates
to infer the predicates should be considered for HIve CBO.
I assume there is no such issue for Calcite instances.

> Apply rule [HiveJoinPushTransitivePredicatesRule] is getting stuck when there are huge
number of predicates 
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-19101
>                 URL: https://issues.apache.org/jira/browse/HIVE-19101
>             Project: Hive
>          Issue Type: Bug
>          Components: CBO
>    Affects Versions: 2.1.1, 2.2.0, 2.3.0, 2.3.1, 2.3.2
>            Reporter: Ganesha Shreedhara
>            Priority: Major
>         Attachments: queries
>
>
> Hive query is getting stuck during the optimisation phase while applying HiveJoinPushTransitivePredicatesRule
when there are huge number of predicates.
>  
> *DEBUG Log:*
> {code:java}
> 2018-04-04T11:22:47,991 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0
main] plan.RelOptPlanner: call#10963: Apply rule [ReduceExpressionsRule(Join)] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1,
$73),joinType=inner,algorithm=none,cost=not available)]
> 2018-04-04T11:22:48,359 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0
main] plan.RelOptPlanner: call#10964: Apply rule [HiveJoinAddNotNullRule] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1,
$73),joinType=inner,algorithm=none,cost=not available)]
> 2018-04-04T11:22:48,360 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0
main] plan.RelOptPlanner: call#10965: Apply rule [HiveJoinPushTransitivePredicatesRule] to
[rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1, $73),joinType=inner,algorithm=none,cost=not
available)]{code}
>  
> *Thread Status:*
>  
> {code:java}
> "6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main" prio=5 tid=0x00007ff18e006800 nid=0x1c03
runnable [0x0000700008176000]
>  java.lang.Thread.State: RUNNABLE
>  at java.util.Arrays.copyOfRange(Arrays.java:2694)
>  at java.lang.String.<init>(String.java:203)
>  at java.lang.StringBuilder.toString(StringBuilder.java:405)
>  at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:95)
>  at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
>  at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:84)
>  at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
>  at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.infer(HiveRelMdPredicates.java:516)
>  at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.inferPredicates(HiveRelMdPredicates.java:426)
>  at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates.getPredicates(HiveRelMdPredicates.java:186)
>  at GeneratedMetadataHandler_Predicates.getPredicates_$(Unknown Source)
>  at GeneratedMetadataHandler_Predicates.getPredicates(Unknown Source)
>  at org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:721)
>  at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinPushTransitivePredicatesRule.onMatch(HiveJoinPushTransitivePredicatesRule.java:83)
>  at org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:314)
>  at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:502)
>  at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:381)
>  at org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:275)
>  at org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:72)
>  at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:206)
>  at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:193)
>  at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:1575)
>  at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1448)
>  at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1174)
>  at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1096)
>  at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113)
>  at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:997)
>  at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
>  at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
>  at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:905)
>  at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:920)
>  at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:330)
>  at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11206)
>  at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:251)
>  at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:265)
>  at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:527)
>  at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)
>  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1499)
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1268)
>  at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>  at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:213)
>  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:447)
>  at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:924)
>  at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:848)
>  at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:735)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
>  at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
>  at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Locked ownable synchronizers:
>  - <0x00000007d5cfcab0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {code}
>  
>  
> This issue seems to be  because of optimisation done on CBO (Specifically on calcite
transitive predicate inference rule) in hive 2.1.1 and later versions. 
> This issue is not there in hive1.2.1 version.
>  
> The queries to reproduce this issue is in this attachment : [^queries]. 
> There is a view (test_view) created by combining the results from 16 tables using UNION
ALL. 
> The query is getting stuck when I run a select query which performs join between view
(test_view) and a table (test1). 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message