Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Mon, 18 Nov 2013 17:31:22 +0000 (UTC)
From: "Prasanth J (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <JIRA.12670659.1380184706129.91453.1384795882896@arcas>
In-Reply-To: <JIRA.12670659.1380184706129@arcas>
References: <JIRA.12670659.1380184706129@arcas>
Subject: [jira] [Updated] (HIVE-5369) Annotate hive operator tree with
 statistics from metastore
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/HIVE-5369?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:all-tabpanel ]

Prasanth J updated HIVE-5369:
-----------------------------

    Attachment: HIVE-5369.10.patch

Fixed the failing test which was recently added.

> Annotate hive operator tree with statistics from metastore
> ----------------------------------------------------------
>
>                 Key: HIVE-5369
>                 URL: https://issues.apache.org/jira/browse/HIVE-5369
>             Project: Hive
>          Issue Type: New Feature
>          Components: Query Processor, Statistics
>    Affects Versions: 0.13.0
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>              Labels: statistics
>             Fix For: 0.13.0
>
>         Attachments: HIVE-5369.1.txt, HIVE-5369.10.patch, HIVE-5369.2.WIP=
.txt, HIVE-5369.2.patch.txt, HIVE-5369.3.patch.txt, HIVE-5369.4.patch.txt, =
HIVE-5369.5.patch.txt, HIVE-5369.6.patch.txt, HIVE-5369.7.patch.txt, HIVE-5=
369.8.patch.txt, HIVE-5369.9.patch, HIVE-5369.9.patch.txt, HIVE-5369.WIP.tx=
t, HIVE-5369.refactor.WIP.txt
>
>
> Currently the statistics gathered at table/partition level and column lev=
el are not used during query planning stage. Statistics at table/partition =
and column level can be used for optimizing the query plans. Basic statisti=
cs like uncompressed data size can be used for better reducer estimation. O=
ther statistics like number of rows, distinct values of columns, average le=
ngth of columns etc. can be used by Cost Based Optimizer (CBO) for making b=
etter query plan selection. As a first step in improving query planning the=
 statistics that are available in the metastore should be attached to hive =
operator tree. The operator tree should be walked and annotated with statis=
tics information. The attached statistics will vary for each operator depen=
ding on the operation it performs. For example, select operator will change=
 the average row size but doesn't affect the number of rows. Similarly filt=
er operator will change the number of rows but doesn't change the average r=
ow size. Similar rules can be applied for other operators as well.=20
> Rules for different operators are added as comments in the code. For more=
 detailed information, the reference book that I am using is "Database Syst=
ems: The Complete Book" by Garcia-Molina et.al.


--
This message was sent by Atlassian JIRA
(v6.1#6144)