hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasanth Jayachandran <pjayachand...@hortonworks.com>
Subject Re: High performance Count Distinct - NO Error
Date Wed, 06 Aug 2014 18:24:47 GMT
Keep an eye on https://issues.apache.org/jira/browse/HIVE-7402 for probabilistic count distinct
UDAF.  Prototype implementation is here https://github.com/t3rmin4t0r/hive-hll-udf/
This uses HyperLogLog++ probabilistic algorithm for count distinct estimation. It has almost
no error for small number of records (in the order of hundreds or few thousands) since it
uses linear counting for
low cardinality. And has a much lesser error for large number of records (mostly <3% for
order of millions or billions of records).

If you would like to know the exact error for your specific dataset then you might need to
use https://github.com/prasanthj/hyperloglog (the prototype implementation mentioned above
uses this but I don’t think it will print the error) cli tool. 
And yes, it uses very low memory when compared to full and accurate count distinct.

Thanks
Prasanth Jayachandran

On Aug 6, 2014, at 1:52 AM, Natarajan, Prabakaran 1. (NSN - IN/Bangalore) <prabakaran.1.natarajan@nsn.com>
wrote:

> Hi
>  
> I am looking for high performance count distinct solution on Hive Query.
>  
> Regular count distinct is very slow but if I use probabilistic count distinct has more
error percentage (if the number of records are small).
>  
>  
> Is there is any solution to have exact count distinct but using low memory and without
error?
>  
> Thanks and Regards
> Prabakaran.N 


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message