hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-17958) spark_dynamic_partition_pruning.q fails when hive.tez.dynamic.semijoin.reduction is false
Date Wed, 01 Nov 2017 21:45:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-17958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16234789#comment-16234789
] 

Sahil Takiar commented on HIVE-17958:
-------------------------------------

Actually {{RedundantDynamicPruningConditionsRemoval}} is probably doing the right thing here.
You don't want the above query to trigger DPP twice because there is a static partition filter
on {{srcpart.hr}}. So maybe {{RedundantDynamicPruningConditionsRemoval}} should only be disabled
if {{hive.tez.dynamic.semijoin.reduction}} is {{true}} and {{hive.execution.engine=tez}}.

> spark_dynamic_partition_pruning.q fails when hive.tez.dynamic.semijoin.reduction is false
> -----------------------------------------------------------------------------------------
>
>                 Key: HIVE-17958
>                 URL: https://issues.apache.org/jira/browse/HIVE-17958
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>            Priority: Major
>
> Looks like {{RedundantDynamicPruningConditionsRemoval}} causes DPP to be disabled in
a few cases (not sure why). When {{hive.tez.dynamic.semijoin.reduction}} is {{true}} (the
default), then this rule is disabled so the normal tests don't hit this issue.
> But when I disable {{hive.tez.dynamic.semijoin.reduction}} then the following query no
longer fully triggers DPP:
> {code}
> EXPLAIN select count(*) from srcpart join srcpart_date on (srcpart.ds = srcpart_date.ds)
join srcpart_hour on (srcpart.hr = srcpart_hour.hr)
> 5777 where srcpart_date.`date` = '2008-04-08' and srcpart_hour.hour = 11 and srcpart.hr
= 11
> {code}
> There should be two DPP sinks, but when the config is set to false, there is only one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message