Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
MIME-Version: 1.0
In-Reply-To: <9b3ad081-557c-8d6c-22fd-ee0ee16a4011@mixmax.com>
References: <9b3ad081-557c-8d6c-22fd-ee0ee16a4011@mixmax.com>
From: David Capwell <dcapwell@gmail.com>
Date: Fri, 8 Dec 2017 19:08:58 -0800
Message-ID: <CAB=riaCfJRibA8X2Mua+Tt2eYU6C8boELpv6_taqyKZrYpskNQ@mail.gmail.com>
Subject: Re: Roaring Bitmap UDFs
To: user@hive.apache.org
Content-Type: multipart/alternative; boundary="089e082212c84d7470055fdf9f02"
archived-at: Sat, 09 Dec 2017 03:09:04 -0000

--089e082212c84d7470055fdf9f02
Content-Type: text/plain; charset="UTF-8"

Think bloom filter that's more dynamic.  It works well when cardinality is
low, but grows quickly to out cost bloom filter as cardinality grows.

This data structure supports existence queries, but your email sounds like
you want count.  If so not really the best fit.

On Dec 8, 2017 5:00 PM, "Nitin Vijayvargiya" <nitinvijay94@gmail.com> wrote:

Hi all,

I'm working on speeding up distinct count calculations, and it looks like
roaring bitmaps (RB) is the newest and meanest way for set operations.
Anyone here have experience with them? How was the performance compared to
hyperloglog and EWAH? A quick google search showed me that it's easier to
find UDF implementations of hyperloglog in presto and hive, but if the hype
is real, it might be worth spending the time to incorporate RB. Also, if
anyone can point me to reliable implementations of UDFs using RB, I would
love to check it out and test it myself =)

Happy Holidays!

Nitin

--089e082212c84d7470055fdf9f02
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"auto"><div>Think bloom filter that&#39;s more dynamic.=C2=A0 It=
 works well when cardinality is low, but grows quickly to out cost bloom fi=
lter as cardinality grows.</div><div dir=3D"auto"><br></div><div dir=3D"aut=
o">This data structure supports existence queries, but your email sounds li=
ke you want count.=C2=A0 If so not really the best fit.<br><div class=3D"gm=
ail_extra" dir=3D"auto"><br><div class=3D"gmail_quote">On Dec 8, 2017 5:00 =
PM, &quot;Nitin Vijayvargiya&quot; &lt;<a href=3D"mailto:nitinvijay94@gmail=
.com">nitinvijay94@gmail.com</a>&gt; wrote:<br type=3D"attribution"><blockq=
uote class=3D"quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;=
padding-left:1ex"> <u></u>        <div style=3D"word-wrap:normal;word-break=
:break-word">   <table class=3D"m_6460327470722575810container" lang=3D"con=
tainer" dir=3D"ltr" border=3D"0" cellpadding=3D"0" cellspacing=3D"0" valign=
=3D"top" style=3D"width:100%"> <tbody><tr> <td valign=3D"top" class=3D"m_64=
60327470722575810message-wrapper" style=3D"color:#222;font-family:arial,san=
s-serif">  <div>Hi all,</div><div><br></div><div>I&#39;m working on speedin=
g up distinct count calculations, and it looks like roaring bitmaps (RB) is=
 the newest and meanest way for set operations. Anyone here have experience=
 with them? How was the performance compared to hyperloglog and EWAH? A qui=
ck google search showed me that it&#39;s easier to find UDF implementations=
 of hyperloglog in presto and hive, but if the hype is real, it might be wo=
rth spending the time to incorporate RB. Also, if anyone can point me to re=
liable implementations of UDFs using RB, I would love to check it out and t=
est it myself =3D)</div><div><br></div><div>Happy Holidays!</div><div><br><=
/div><div>Nitin</div><img align=3D"left" width=3D"0" height=3D"0" style=3D"=
border:0;width:0px;height:0px" src=3D"https://track.mixmax.com/api/track/v2=
/A5yu0xg8dziGmPmvW/i02bj5CbpFWbnBEN5kXYqlmdulGdp5mI/gInJ3buUGajFGch5SZ2lGaA=
JXZzVnI/?sc=3Dfalse" alt=3D"">  </td> </tr> </tbody></table> </div>  </bloc=
kquote></div><br></div></div></div>

--089e082212c84d7470055fdf9f02--