hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vishesh Garg (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-12989) Hive with ORC - Data read from disk for 2 queries same despite very different selectivities
Date Wed, 03 Feb 2016 05:07:39 GMT
Vishesh Garg created HIVE-12989:
-----------------------------------

             Summary: Hive with ORC - Data read from disk for 2 queries same despite very
different selectivities
                 Key: HIVE-12989
                 URL: https://issues.apache.org/jira/browse/HIVE-12989
             Project: Hive
          Issue Type: Bug
          Components: ORC
    Affects Versions: 1.2.1
         Environment: TPCDS 100 GB
            Reporter: Vishesh Garg
            Priority: Minor


I'm using Hive 1.2.1 with ORC file format and doing some profiling for the data read from
the disk (after clearing file system caches) for 2 queries, the data being loaded from the
TPCDS benchmark. The queries are:

{code:sql}
 Q1) select count(ss_item_sk) from store_sales where ss_sold_date_sk between 2450816 and 2450817;
 Q2) select  count(ss_item_sk) from store_sales where ss_sold_date_sk between 2450816 and
2452000;
{code}

I'm using flags *hive.vectorized.execution.enabled* = true and *hive.optimize.index.filter*
= true.

The output counts are about .1M and about 117M for Q1 and Q2 resp. On looking at the hive
logs, I see that about 11000 row-groups are included for Q1 and 33000 row-groups are included
for Q2 after filtering using index. Hence, I had expected vastly different sizes of data read.
However, I see about 1GB of data read for both the queries. I suspect this has something to
do with the function _addRgFilteredStreamToRanges()_



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message