hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Chauhan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-9341) Apply ColumnPrunning for noop PTFs
Date Sat, 24 Jan 2015 00:17:35 GMT

     [ https://issues.apache.org/jira/browse/HIVE-9341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ashutosh Chauhan updated HIVE-9341:
-----------------------------------
       Resolution: Fixed
    Fix Version/s: 0.15.0
           Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Navis!

> Apply ColumnPrunning for noop PTFs
> ----------------------------------
>
>                 Key: HIVE-9341
>                 URL: https://issues.apache.org/jira/browse/HIVE-9341
>             Project: Hive
>          Issue Type: Improvement
>          Components: PTF-Windowing
>            Reporter: Navis
>            Assignee: Navis
>            Priority: Trivial
>             Fix For: 0.15.0
>
>         Attachments: HIVE-9341.1.patch.txt, HIVE-9341.2.patch.txt, HIVE-9341.3.patch.txt,
HIVE-9341.4.patch.txt
>
>
> Currently, PTF disables CP optimization, which can make a huge burden. For example,
> {noformat}
> select p_mfgr, p_name, p_size,
> rank() over (partition by p_mfgr order by p_name) as r,
> dense_rank() over (partition by p_mfgr order by p_name) as dr,
> sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding
and current row) as s1
> from noop(on part 
>   partition by p_mfgr
>   order by p_name
>   );
> STAGE PLANS:
>   Stage: Stage-1
>     Map Reduce
>       Map Operator Tree:
>           TableScan
>             alias: part
>             Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats:
NONE
>             Reduce Output Operator
>               key expressions: p_mfgr (type: string), p_name (type: string)
>               sort order: ++
>               Map-reduce partition columns: p_mfgr (type: string)
>               Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats:
NONE
>               value expressions: p_partkey (type: int), p_name (type: string), p_mfgr
(type: string), p_brand (type: string), p_type (type: string), p_size (type: int), p_container
(type: string), p_retailprice (type: double), p_comment (type: string), BLOCK__OFFSET__INSIDE__FILE
(type: bigint), INPUT__FILE__NAME (type: string), ROW__ID (type: struct<transactionid:bigint,bucketid:int,rowid:bigint>)
> ...
> {noformat}
> There should be a generic way to discern referenced columns but before that, we know
CP can be safely applied to noop functions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message