hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth J (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-5369) Annotate hive operator tree with statistics from metastore
Date Thu, 14 Nov 2013 21:19:22 GMT

     [ https://issues.apache.org/jira/browse/HIVE-5369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Prasanth J updated HIVE-5369:
-----------------------------

    Attachment: HIVE-5369.7.patch.txt

Regenerated golden files for most of the qfiles that contain explain extended.

> Annotate hive operator tree with statistics from metastore
> ----------------------------------------------------------
>
>                 Key: HIVE-5369
>                 URL: https://issues.apache.org/jira/browse/HIVE-5369
>             Project: Hive
>          Issue Type: New Feature
>          Components: Query Processor, Statistics
>    Affects Versions: 0.13.0
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>              Labels: statistics
>             Fix For: 0.13.0
>
>         Attachments: HIVE-5369.1.txt, HIVE-5369.2.WIP.txt, HIVE-5369.2.patch.txt, HIVE-5369.3.patch.txt,
HIVE-5369.4.patch.txt, HIVE-5369.5.patch.txt, HIVE-5369.6.patch.txt, HIVE-5369.7.patch.txt,
HIVE-5369.WIP.txt, HIVE-5369.refactor.WIP.txt
>
>
> Currently the statistics gathered at table/partition level and column level are not used
during query planning stage. Statistics at table/partition and column level can be used for
optimizing the query plans. Basic statistics like uncompressed data size can be used for better
reducer estimation. Other statistics like number of rows, distinct values of columns, average
length of columns etc. can be used by Cost Based Optimizer (CBO) for making better query plan
selection. As a first step in improving query planning the statistics that are available in
the metastore should be attached to hive operator tree. The operator tree should be walked
and annotated with statistics information. The attached statistics will vary for each operator
depending on the operation it performs. For example, select operator will change the average
row size but doesn't affect the number of rows. Similarly filter operator will change the
number of rows but doesn't change the average row size. Similar rules can be applied for other
operators as well. 
> Rules for different operators are added as comments in the code. For more detailed information,
the reference book that I am using is "Database Systems: The Complete Book" by Garcia-Molina
et.al.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message