phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-4164) APPROX_COUNT_DISTINCT becomes imprecise at 20m unique values.
Date Wed, 06 Sep 2017 05:09:00 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154817#comment-16154817
] 

Lars Hofhansl commented on PHOENIX-4164:
----------------------------------------

Since the correctness drops of steeply, I'm guessing we're running into precision limits of
the hash values, and it might simply be a task of increasing that.

I think we have three options:
# Increase the default precision. And _state_ how many unique value we expect to be able to
could with that.
# Make the precision configurable (for the record I do not think that's a good idea)
# Leave the precision as is, but still state how unique value this should support.



> APPROX_COUNT_DISTINCT becomes imprecise at 20m unique values.
> -------------------------------------------------------------
>
>                 Key: PHOENIX-4164
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4164
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>
> {code}
> 0: jdbc:phoenix:localhost> select count(*) from test;
> +-----------+
> | COUNT(1)  |
> +-----------+
> | 26931816  |
> +-----------+
> 1 row selected (14.604 seconds)
> 0: jdbc:phoenix:localhost> select approx_count_distinct(v1) from test;
> +----------------------------+
> | APPROX_COUNT_DISTINCT(V1)  |
> +----------------------------+
> | 17221394                   |
> +----------------------------+
> 1 row selected (21.619 seconds)
> {code}
> The table is generated from random numbers, and the cardinality of v1 is close to the
number of rows.
> (I cannot run a COUNT(DISTINCT(v1)), as it uses up all memory on my machine and eventually
kills the regionserver - that's another story and another jira)
> [~aertoria]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message