hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neil Best (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-10419) can't do query on partitioned view with analytic function in strictmode
Date Thu, 23 Jun 2016 15:41:16 GMT

    [ https://issues.apache.org/jira/browse/HIVE-10419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346621#comment-15346621
] 

Neil Best commented on HIVE-10419:
----------------------------------

It appears to me that using an analytic function in a view defeats partition pruning of the
underlying table somehow.  Consider the following which may be helpful:

{noformat}
0: jdbc:hive2://pco-ph-gbdm-03:10000> set hive.mapred.mode;
+-----------------------------+--+
|             set             |
+-----------------------------+--+
| hive.mapred.mode=nonstrict  |
+-----------------------------+--+
0: jdbc:hive2://pco-ph-gbdm-03:10000> drop view if exists sla.minute_order_v;
No rows affected (0.105 seconds)
0: jdbc:hive2://pco-ph-gbdm-03:10000> 
0: jdbc:hive2://pco-ph-gbdm-03:10000> create view if not exists sla.minute_order_v
0: jdbc:hive2://pco-ph-gbdm-03:10000> as select
0: jdbc:hive2://pco-ph-gbdm-03:10000>     flight_id,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     tail,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     acpu_time,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     device,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     alt_m,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     agl_ft,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     row_number() over (
0: jdbc:hive2://pco-ph-gbdm-03:10000>         partition by flight_source, flight_id, device
0: jdbc:hive2://pco-ph-gbdm-03:10000>     order by acpu_time) asc_order,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     row_number() over (
0: jdbc:hive2://pco-ph-gbdm-03:10000>         partition by flight_source, flight_id, device
0: jdbc:hive2://pco-ph-gbdm-03:10000>     order by acpu_time desc) desc_order,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     year,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     month,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     day,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     flight_source
0: jdbc:hive2://pco-ph-gbdm-03:10000> from
0: jdbc:hive2://pco-ph-gbdm-03:10000>     sla.minute
0: jdbc:hive2://pco-ph-gbdm-03:10000> where
0: jdbc:hive2://pco-ph-gbdm-03:10000>     agl_ft >= 10000
0: jdbc:hive2://pco-ph-gbdm-03:10000>     and device = 'sla';
No rows affected (0.103 seconds)
0: jdbc:hive2://pco-ph-gbdm-03:10000> explain authorization select * from minute_order_v
where year = 2016 and month = 5 and day = 23 and flight_source = 'gdw';
+--------------------------------------------------------------------------------------------------------------------------------------------+--+
|                                                                  Explain               
                                                   |
+--------------------------------------------------------------------------------------------------------------------------------------------+--+
| INPUTS:                                                                                
                                                   |
|   sla@minute_order_v                                                                   
                                                   |
|   sla@minute                                                                           
                                                   |
|   sla@minute@year=2016/month=5/day=23/flight_source=gdw                                
                                                   |
|   sla@minute@year=2016/month=5/day=23/flight_source=periodic                           
                                                   |
|   sla@minute@year=2016/month=5/day=24/flight_source=gdw                                
                                                   |
|   sla@minute@year=2016/month=5/day=24/flight_source=periodic                           
                                                   |
| OUTPUTS:                                                                               
                                                   |
|   hdfs://nameservice1/tmp/hive/etl-user/0d2d8c59-1dbe-4fb0-8cde-478ff33841de/hive_2016-06-23_15-02-16_343_5980077610772883275-6/-mr-10000
 |
| CURRENT_USER:                                                                          
                                                   |
|   etl-user                                                                             
                                                   |
| OPERATION:                                                                             
                                                   |
|   CREATEVIEW                                                                           
                                                   |
| AUTHORIZATION_FAILURES:                                                                
                                                   |
|   No privilege 'Select' found for inputs { database:sla, table:minute_order_v}         
                                                   |
|   No privilege 'Select' found for inputs { database:sla, table:minute}                 
                                                   |
+--------------------------------------------------------------------------------------------------------------------------------------------+--+
16 rows selected (0.269 seconds)
0: jdbc:hive2://pco-ph-gbdm-03:10000> drop view if exists sla.minute_order_v;
No rows affected (0.07 seconds)
0: jdbc:hive2://pco-ph-gbdm-03:10000> 
0: jdbc:hive2://pco-ph-gbdm-03:10000> create view if not exists sla.minute_order_v
0: jdbc:hive2://pco-ph-gbdm-03:10000> as select
0: jdbc:hive2://pco-ph-gbdm-03:10000>     flight_id,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     tail,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     acpu_time,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     device,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     alt_m,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     agl_ft,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     -- row_number() over (
0: jdbc:hive2://pco-ph-gbdm-03:10000>     --     partition by flight_source, flight_id,
device
0: jdbc:hive2://pco-ph-gbdm-03:10000>     -- order by acpu_time) asc_order,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     -- row_number() over (
0: jdbc:hive2://pco-ph-gbdm-03:10000>     --     partition by flight_source, flight_id,
device
0: jdbc:hive2://pco-ph-gbdm-03:10000>     -- order by acpu_time desc) desc_order,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     year,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     month,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     day,
0: jdbc:hive2://pco-ph-gbdm-03:10000>     flight_source
0: jdbc:hive2://pco-ph-gbdm-03:10000> from
0: jdbc:hive2://pco-ph-gbdm-03:10000>     sla.minute
0: jdbc:hive2://pco-ph-gbdm-03:10000> where
0: jdbc:hive2://pco-ph-gbdm-03:10000>     agl_ft >= 10000
0: jdbc:hive2://pco-ph-gbdm-03:10000>     and device = 'sla';
No rows affected (0.097 seconds)
0: jdbc:hive2://pco-ph-gbdm-03:10000> explain authorization select * from minute_order_v
where year = 2016 and month = 5 and day = 23 and flight_source = 'gdw';
+--------------------------------------------------------------------------------------------------------------------------------------------+--+
|                                                                  Explain               
                                                   |
+--------------------------------------------------------------------------------------------------------------------------------------------+--+
| INPUTS:                                                                                
                                                   |
|   sla@minute_order_v                                                                   
                                                   |
|   sla@minute                                                                           
                                                   |
|   sla@minute@year=2016/month=5/day=23/flight_source=gdw                                
                                                   |
| OUTPUTS:                                                                               
                                                   |
|   hdfs://nameservice1/tmp/hive/etl-user/0d2d8c59-1dbe-4fb0-8cde-478ff33841de/hive_2016-06-23_15-02-51_565_9081433377804806084-6/-mr-10000
 |
| CURRENT_USER:                                                                          
                                                   |
|   etl-user                                                                             
                                                   |
| OPERATION:                                                                             
                                                   |
|   CREATEVIEW                                                                           
                                                   |
| AUTHORIZATION_FAILURES:                                                                
                                                   |
|   No privilege 'Select' found for inputs { database:sla, table:minute_order_v}         
                                                   |
|   No privilege 'Select' found for inputs { database:sla, table:minute}                 
                                                   |
+--------------------------------------------------------------------------------------------------------------------------------------------+--+
13 rows selected (0.233 seconds)
{noformat}

This was the best place I could find to report this.  Please advise if you know of something
better.

> can't do query on partitioned view with analytic function in strictmode
> -----------------------------------------------------------------------
>
>                 Key: HIVE-10419
>                 URL: https://issues.apache.org/jira/browse/HIVE-10419
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive, Views
>    Affects Versions: 0.13.0, 0.14.0, 1.0.0
>         Environment: Cloudera 5.3.x. 
>            Reporter: Hector Lagos
>
> Hey Guys,
> I created the following table:
> CREATE TABLE t1 (id int, key string, value string) partitioned by (dt int);
> And after that i created a view on that table as follow:
> create view v1 PARTITIONED ON (dt)
> as
> SELECT * FROM (
> SELECT row_number() over (partition by key order by value asc) as row_n, * FROM t1 
> ) t WHERE row_n = 1;
> We are working with hive.mapred.mode=strict and when I try to do the  query select *
from v1 where dt = 2 , I'm getting the following error:
> FAILED: SemanticException [Error 10041]: No partition predicate found for Alias "v1:t:t1"
Table "t1"
> Is this a bug or a limitation of Hive when you use analytic functions in partitioned
views? If i remove the row_number function it works without problems. 
> Thanks in advance, any help will be appreciated. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message