pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4767) Partition filter not pushed down when filter clause references variable from another load path
Date Wed, 12 Jul 2017 21:42:00 GMT

    [ https://issues.apache.org/jira/browse/PIG-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084758#comment-16084758
] 

Daniel Dai commented on PIG-4767:
---------------------------------

That's right, PartitionFilterOptimizer and PredicatePushdownOptimizer does not push filter
up. The problem PIG-1669 try to solve does not exist. +1.

> Partition filter not pushed down when filter clause references variable from another
load path
> ----------------------------------------------------------------------------------------------
>
>                 Key: PIG-4767
>                 URL: https://issues.apache.org/jira/browse/PIG-4767
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.15.0
>            Reporter: Anthony Hsu
>            Assignee: Koji Noguchi
>             Fix For: 0.18.0
>
>         Attachments: pig-4767-v01.patch
>
>
> To reproduce:
> {noformat:title=test.pig}
> a = load 'a.txt';
> a_group = group a all;
> a_count = foreach a_group generate COUNT(a) as count;
> b = load 'mytable' using org.apache.hcatalog.pig.HCatLoader();
> b = filter b by datepartition == '2015-09-01-00' and foo == a_count.count;
> dump b;
> {noformat}
> The above query ends up reading all the table partitions. If you remove the {{foo ==
a_count.count}} clause or replace {{a_count.count}} with a constant, then partition filtering
happens properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message