hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-9689) Store distinct value estimator's bit vectors in metastore
Date Sun, 22 Mar 2015 17:49:10 GMT

     [ https://issues.apache.org/jira/browse/HIVE-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Prasanth Jayachandran updated HIVE-9689:
----------------------------------------
    Description: Hive currently uses PCSA (Probabilistic Counting and Stochastic Averaging)
algorithm to determine distinct cardinality. The NDV value determined from the UDF is stored
in the metastore instead of the actual bit vectors. This makes it impossible to estimate the
overall NDV across all the partition (or selected partitions). We should ideally store the
bitvectors in the metastore and do server side merging of the bitvectors. Also we could replace
the current PCSA algorithm in favour of HyperLogLog if space is a constraint.   (was: Hive
currently uses PCSA (Probabilistic Counting and Stochastic Averaging) algorithm to determine
distinct cardinality. The NDV value determined from the UDF is stored in the metastore instead
of the actual bit vectors. This makes it impossible to estimation the overall NDV across all
the partition (or selected partitions). We should ideally store the bitvectors in the metastore
and do server side merging of the bitvectors. Also we could replace the current PCSA algorithm
in favour of HyperLogLog if space is a constraint. )

> Store distinct value estimator's bit vectors in metastore
> ---------------------------------------------------------
>
>                 Key: HIVE-9689
>                 URL: https://issues.apache.org/jira/browse/HIVE-9689
>             Project: Hive
>          Issue Type: New Feature
>            Reporter: Prasanth Jayachandran
>              Labels: gsoc2015
>
> Hive currently uses PCSA (Probabilistic Counting and Stochastic Averaging) algorithm
to determine distinct cardinality. The NDV value determined from the UDF is stored in the
metastore instead of the actual bit vectors. This makes it impossible to estimate the overall
NDV across all the partition (or selected partitions). We should ideally store the bitvectors
in the metastore and do server side merging of the bitvectors. Also we could replace the current
PCSA algorithm in favour of HyperLogLog if space is a constraint. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message