hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashish Thusoo (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HIVE-218) predicate on partitioning column is ignored in many places
Date Tue, 10 Mar 2009 22:27:50 GMT

     [ https://issues.apache.org/jira/browse/HIVE-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ashish Thusoo updated HIVE-218:
-------------------------------

             Priority: Critical  (was: Blocker)
    Affects Version/s: 0.3.0

Marking this as a critical as there is a workaround for this by putting the predicate on the
partitioning column within a subquery.

In the example above, this can be achieved by

FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 .... JOIN (SELECT aaa.* FROM
aaa WHERE aaa.ds-2009-01-05) e ....


> predicate on partitioning column is ignored in many places
> ----------------------------------------------------------
>
>                 Key: HIVE-218
>                 URL: https://issues.apache.org/jira/browse/HIVE-218
>             Project: Hadoop Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.3.0
>            Reporter: Joydeep Sen Sarma
>            Assignee: Prasad Chakka
>            Priority: Critical
>
> We tried two queries yesterday that bought up several problems:
> 1. predicate on partitioning column within a join clause was ignored:
> FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM yyy SELECT
b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT c.xx, c.yy, c.ds WHERE
c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx AND e.ds=2009-01-05) INSERT OVERWRITE TABLE ...
> the plan tried to scan *all* partitions!
> 2. predicate on partitioning clause inside insert clause was ignored (we took the previous
query and moved the partition filter to the insert statement)
> FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM yyy SELECT
b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT c.xx, c.yy, c.ds WHERE
c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx ) INSERT OVERWRITE TABLE ... WHERE e.ds=2009-01-05;

> the plan again tried to scan *all* partitions
> the really bad thing is that we were able to detect this problem only because of metastore
inconsistencies - otherwise - we would have merrily scanned all the data. This is really critical
to get fixed - because this means that we may actually be scanning tons of unnecessary data
in production.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message